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INTRODUCTION 

1 . 1  Nature  of  the  Problem 

Routine  mammograms  are  an  essential  element  of  health  care  for 
adult  women  (1-5)  .  Digital  technology  promises  improved  lesion 
detection  and,  in  conjunction  with  teleradiology,  improved  access 
to  mammography  compared  to  current  conventional  techniques  (6,7). 
However,  because  mammography  requires  high  spatial  resolution  over 
a  large  area,  good  contrast  resolution  over  a  large  dynamic  range, 
and  high  detective  efficiency  for  x-rays,  the  development  of  full- 
breast  digital  imaging  (FBDI)  technology  has  been  slow  (reviewed  in 
8) .  While  FBDI  technology  will  be  implemented  over  the  next  three 
to  five  years,  the  current  screen  film  mammography  examinations 
will  remain  the  standard  for  use  in  teleradiology  systems.  Digital 
imaging  methods  will  ultimately  replace  analog  (screen-film) 
systems;  filmless  systems  will  become  an  integral  part  of  local- 
area  and  wide-area  networks  and  digital  x-ray  mammography  units 
will  eventually  reach  the  market.  These  modalities  generate  large 
volumes  of  digital  data  and  thus  will  require  an  efficient 
communications  interface  if  the  data  are  to  be  managed  in 
reasonable  time  frames.  Furthermore,  a  standard  interface  must  be 
implemented  lest  the  telemammography  benefits  envisioned  for  the 
near  future  disappear  into  a  costly  and  inefficient  world  of 
incompatible  systems.  We  believe  it  is  essential  now,  during  the 
prototype  stage  of  system  development,  to  implement  and  evaluate  an 
interface  that  satisfies  the  Digital  Imaging  and  Communications  in 
Medicine  (DICOM  3.0)  standard  (9-13). 
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To  make  transmission  and  archival  of  digital  image  files 
practically  feasible,  image  compression  algorithms  (14-16)  are 
necessary  as  a  means  of  compacting  and  reducing  the  enormous  data 
sets  generated  by  a  digital  full -breast  mammogram  (4K  x  4K  x  2 
Bytes  =  32  MB/image  x  4  images/screening  exam,  or  8  images  for 
comparing  past  and  present  exams) .  Using  software  (AWARE) ,  Inc, 
Waltham,  MA)  for  wavelet  compression  (17-23) ,  we  have  studied 
wavelet  transforms  of  digitized  screen- film  mammograms  at 
compression  ratios  of  50:1  to  100:1;  and  have  determined  from  a 
contrast -detail  phantom  study  that  50:1  is  an  acceptable  ratio. 
Wavelet  transforms  are  multiresolution  decomposition  tools,  with 
the  kernel  of  wavelet  transform  obtained  by  dilation  and 
translation  of  a  selected  bandpass  filter.  Compression  for 
digitally  acquired  images  will  demand  new  filter  designs  to 
attenuate  the  effects  from  the  high  detective  quantum  efficiency 
and  low  noise  of  digital  detectors  without  distorting  mammographic 
features . 

The  use  of  T1  digital  transmission  service  (1.544  Mbits/sec 
signaling  speed)  is  expensive  for  use  in  telemammography.  An 
estimate  of  T1  costs  is  $19/month/mile  per  site  and  access  charges 
of  approximately  $1,000  per  month.  The  use  of  a  successful 
compression  ratio  of  24  to  1  enables  a  DS-0  digital  service  (64K 
bits/sec)  to  accomplish  the  same  signaling  rate  of  a  DS-1 
(equivalent  to  T1  service)  channel. 

Digital  display  technologies  are  limited,  for  the  near  future, 
to  8/12  bit  pixel  laser  film  printers  and  2K  x  2 . 5K  interactive 
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high-resolution  gray-scale  monitors.  We  have  extensive  experience 
with  both  of  these  display  modalities.  Like  other  components  of 
the  digital  mammography  system,  the  display  modalities  face  special 
challenges  as  a  result  of  the  amount  of  data  generated  by  digitally 
acquired  examinations.  The  critical  effort  with  the  laser  film 
printer  is  the  development  of  appropriate  look-up  tables  (24-27) , 
so  that  the  digital  detector  output,  which  is  a  4K  x  4K  digital 
array  with  63 -micron  pixel  size  and  12  bits/pixel  of  dynamic  range, 
is  optimally  transferred  at  8 /bits  per  pixel  to  the  laser- imaged 
films.  In  the  case  of  the  gray- scale  displays,  the  problems  are 
more  complex  (28-34) .  To  be  usefully  implemented,  there  are  four 
standard  mamraographic  (screening)  views  that  must  be  displayed 
(craniocaudal  (CC)  and  mediolateral  oblique  (MLO)  views  of  each 
breast)  at  a  two-monitor  workstation;  in  some  cases,  at  least  two 
diagnostic  views  (spot  compression  or  magnification) (  performed  in 
two  different  projections  (usually  90°  apart)  will  also  be  viewed, 
as  will  both  the  current  and  the  previous  examination.  A 
mammographer  using  a  gray-scale  monitor  will  need  rapid  displays 
with  easily  used  interactive  commands.  In  addition,  the  system 
must  effectively  display  the  4K  x  4K  image  matrix  on  a  2K  x  2.5K 
screen.  A  well -designed  display  protocol  meeting  all  these  needs 
could  become  a  standard  implemented  by  a  digital  signal  processing 
(DSP)  board  or  an  accelerator  board.  Such  a  standard  would  enable 
faster  throughputs  for  the  mammographer  and  aid  in  the 
establishment  of  quality  control  standards  for  visualizing  digital 
mammography  examinations. 
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1 . 2  Purpose  of  the  Present  Work 


The  research  hypothesis  being  tested  is  that  a  telemammography 
system  can  interpret  mammography  images  with  an  accuracy  level 
sufficient  for  primary  diagnosis  utilizing  a  film  digitizer  at  the 
transmitting  site  (with  a  50 -micron  pixel  size  for  spatial 
resolution  and  12  bit  pixel  range  for  contrast)  and  interactive 
grayscale  display  monitors  (2048  x  2560  x  8/12  bits)  at  the 
receiving  site. 

A  successful  telemammography  system  will  provide  benefits  in 
the  following  four  areas: 

A.  PRIMARY  DIAGNOSIS.  Telemammography  offers  the  ability  to 
provide  mammographic  consultations  to  underserved  and 
remote  areas.  Achieving  the  image  quality  required  of  a 
telemammography  system  for  primary  diagnosis  will  enable 
an  outreach  program  to  enhance  a  region's  breast 
screening  programs  and  to  improve  patient  care. 

B.  INTEGRATION  OF  MAMMOGRAPHY  GROUP  PRACTICE  DISTRIBUTED 

OVER  MULTIPLE  IMAGING  CENTERS.  As  the  awareness 
regarding  the  role  of  mammography  in  early  detection  of 
breast  cancer  increases,  so  does  the  need  for 
accessibility  to  low  cost  screening  mammography.  More 
and  more  practices  are  responding  to  the  rapidly  growing 
utilization  of  mammography  by  opening  satellites  to 
imaging  practices.  Telemammography  would  enable  a  group 
with  a  limited  number  of  expert  mammographers  to  handle 
multiple  off -site  practices.  Additionally,  if 
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appropriate  for  the  practice,  the  radiologist  could 
supervise  screening  mammograms  off-site  and  determine  the 
need  for  any  additional  views  at  the  time  of  the 
examination  instead  of  having  the  patient  return  at  a 
later  time.  Image  quality  could  also  be  supervised  off¬ 
site  via  telemammography.  Another  advantage  of  this 
system  is  that,  due  to  efficiencies  of  scale,  mammography 
costs  would  be  lower  and  a  lower  fee  for  interpretation 
could  be  maintained  without  the  need  for  an  on-site 
radiologist.  In  part,  this  would  be  related  to 
alleviating  the  need  for  the  physician  to  travel  to  and 
from  various  satellite  screening  sites. 

C.  OVERREADING  OF  MAMMOGRAPHY  IMAGES.  There  is  increasing 
emphasis  on  the  interpretive  skills  of  radiologist's 
reading  mammograms  as  part  of  the  quality  assurance 
process  monitored  both  by  the  ACR  Mammography 
Accreditation  program  and  by  the  Food  &  Drug 

Administration.  Residency  programs  are  offering  more 
time  in  mammography  rotations  now  compared  with  only  a 
few  years  ago;  there  have  been  formal  standardized 
training  programs  for  radiology  residents  and 
mammographic  technologists  through  the  ACR-CDC 

Cooperative  Agreement.  Nonetheless,  the  impact  of  the 
accreditation  guidelines  and  the  training  programs  will 
not  be  immediate,  and  there  remains  a  need  for  expert 
mammographic  interpretation  in  many  practices.  With 
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telemammography,  a  small  number  of  expert  mammographers 
could  provide  consultation  services  or  second  readings  of 
mammograms  for  a  larger  number  of  general  radiologists, 
and  improve  the  quality  of  care.  Additionally,  the  data 
and  images  for  patients  in  a  region  could  be  stored  and 
utilized  for  the  development  of  a  regional  mammography 
database . 

D .  IMPROVED  CONSULTATION  WITH  SURGEON  AND  PCP  .  Primary  care 
physicians  and  surgeons  could  review  the  mammograms  on 
their  patients  without  the  need  for  "signing  out"  the 
original  films,  thus  providing  more  reliable  continuing 
service  and  decreasing  the  risk  of  loss  of  films.  On  a 
broader  scale,  the  utilization  of  telemammography  at 
multiple  radiology  practices  in  a  referral  region  could 
provide  greatly  improved  access  to  a  patient's  prior 
examination,  regardless  of  where  the  patient  obtained 
subsequent  mammograms .  The  importance  of  such 

transmission  would  be  multifocal:  original  films  would 
not  need  to  be  mailed,  risking  their  loss,  the  cost  of 
making  copy  films  could  be  avoided:  and  the  facility 
interpreting  the  current  study  would  have  a  much  more 
rapid  access  to  the  prior  exams,  thereby,  improving  the 
accuracy  of  diagnosis  and  diminishing  the  anxiety  of  the 
patient  who  is  waiting  for  her  final  results. 

1.3  Methods  of  Approach 
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Three  tasks  are  required  in  support  of  evaluating  the  research 
hypothesis . 

Task  1 .  A  selected  set  of  analog  mammographic  films  will  be 
collected  and  digitized  using  a  laser  film  digitizer  set  at  a  50- 
raicron  spot  size  and  a  12 -bits  dynamic  range.  An  ROC  analysis  will 
be  conducted  on  the  analog  mammographic  films  and  the  digitized 
films  displayed  on  grayscale  monitors  (2048  x  2560  x  8/12  bits) . 

Task  2 .  A  digital  communication  network  will  be  implemented 
between  the  Department  of  Radiology  Breast  Imaging  Center  in  the 
Diagnostic  Center  for  Women  (primary  Care  Center  Building,  UVa)  and 
the  off-campus  outpatient  Virginia  Mammography  Center  (Northridge 
facility,  UVa,  8  miles  from  the  campus) .  A  laser  film  digitizer 
(50-micron  spot  size.  Model  150,  Lumisys  Inc,  Sunnyvale,  CA)  and 
computer  workstation  (SUN,  SPARC  Model  40)  will  be  installed  at  the 
Northridge.  Transmission  of  the  digitized  mammographic  films  will 
be  over  a  T-1  carrier  (1.544  Mbits/sec  signaling  speed)  to  the 
department's  PACS  and  displayed  on  2048  x  2560  x  8/12-bit  grayscale 
monitors.  A  protocol  for  end-to-end  telemammographic  quality 
control  will  be  implemented. 

Task  3  A  performance  evaluation  will  be  conducted  of  the 
teleradiology  system  using  the  metrics  of  response  time, 
throughput,  reliability,  and  clinical  acceptance. 

We  divided  the  above  three  tasks  into  the  following  aims: 

Aim  1 .  Collection  of  an  adequate  retrospective  database  of 
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analog  raammographic  film  images  and  patient  data  for  use  in 
evaluating  a  telemammography  system. 

Aim  2 .  Convert  the  collected  database  of  analog  mammographic 
films  into  digital  arrays  using  a  laser  film  digitizer  with  a  50 
micron  pixel  spot  size  and  12  bits  per  pixel  of  dynamic  range. 

Aim  3 .  Conduct  an  ROC  analysis  of  the  retrospective  database  of 
the  analog  mammographic  images  and  the  digitized  arrays  displayed 
on  the  2048  x  2560  x  8/12-bit  grayscale  monitors. 

Aim  4 ■  Implement  a  digital  transmission  service  between  the 
Virginia  Mammography  Center  at  Northridge  and  the  PACS  in  the 
University  of  Virginia  Department  of  Radiology  and  its  workstations 
including  that  in  Diagnostic  Center  for  Women. 

Aim  5 .  Design,  implement,  and  evaluate  an  end-to-end  quality 
control  program  for  the  telemammography  system. 

Aim  6 .  A  performance  evaluation  will  be  conducted  of  the 
telemammography  system  using  the  metrics  of  response  time, 
throughput,  reliability,  and  clinical  acceptance. 

2 .  Experimental  Methods  and  Results 
2.1  Statement  of  Work. 

The  proposed  statement  of  work  for  the  contract  was  identified 
by  the  year  and  aim  as  follows.  We  present  these  Tasks  and  Aims, 
commenting  on  our  current  progress  at  the  completion  of  the  first 
year  of  the  contract  (in  script  font) . 

YEAR  1 

TASK  1:  Aim  1 
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*  Complete  collection  of  200  normal  and  200  biopsy-proven 
malignant  analog  mammographic  films  to  form  an  image 
database  (6  months  to  complete) . 

*  Collect  pathology  and  consultation  reports  for 
the  400  images  in  the  database. 

*  Conduct  an  image  quality  control  protocol  on 
the  image  database  to  insure  correct  ground 
truth  identification,  correct  diagnosis,  and 
an  adequate  optical  density  range  in  each 
image . 

*  Conduct  a  review  of  the  identified  Regions  of 
Interest  (ROIs)  to  insure  proper  identificati 
on. 


We  have  completed  the  collection  of  200  normal  and  200 
abnormal  analog  mammographic  films  to  form  our  image  data  base  (see 
Appendix  I  for  a  listing  (coded  by  case  number)  .  (Abnormal  cases 
include  benign  and  malignant  lesions,  with  pathology  serving  as 
gound  truth.)  We  have  completed  the  collection  of  patient  data  and 
will  be  adding  the  patient' s  age  as  well  as  demographic  data.  We 
conducted  an  image  quality  control  protocol  on  the  analog  image 
database  to  insure  correct  ground  truth  identification,  correct 
diagnosis ,  and  the  proper  optical  density  range  in  each  image.  All 
cases  were  reviewed  and  lesions  were  analyzed  and  classified  using 
ACR  lexicon.  The  abnormal  cases  were  also  verified  for  presence  of 
only  one  lesion.  The  abnormals  selected  reflected  a  range  of 
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difficulty  in  lesion  perception  and  analysis . 

Normals  were  selected  as  normal  based  on  the  following :  (1) 

Initial  consultation  reading  was  normal;  (2)  review  of  images 
showed  no  significant  abnormality;  and  (3)  follow-up  mammogram  at 
least  24  months  later  showed  no  interval  change.  Mammographic 
findings  of  intramammory  lymph  nodes,  calcification  of  fat 
necrosis ,  dermal  calcifications  and  vascular  calcification  are 
considered  pathognomically  benign  and  could  be  present  on  "normal" 
cases . 

Parenchymal  density  for  each  case  was  classified  on  a  scale  of 
1  to  4.  (1  =  fatty;  2  =  scattered  fibroglandular  tissue;  3  = 

heterogenously  dense;  and  4  =  extremely  dense)  .  The  density  of 
normal  cases  was  matched  to  abnormals .  There  were  an  approximately 
equal  number  of  fatty  normals,  fatty  abnormals ,  etc. 

The  image  database  was  initially  collected  together  with  an 
overlay  sheet  of  clear  plastic  identifying  the  Regions  of  Interest 
(ROIs)  to  insure  proper  identification.  Our  intent  was  to  digitize 
this  RBI  and  use  it  for  display  on  the  grayscale  workstations .  We 
have  discontinued  this  portion  of  the  project  as  it  introduces  a 
bias  in  the  reader  response.  That  is,  if  only  a  IK  x  IK  ROI  is 
displayed  to  each  reader,  a  bias  is  introduced  by  not  displaying 
the  full  image. 

YEAR  1. 

TASK  1:  Aim  2 

*  Begin  digitization  of  analog  mammographic 
films  (requires  eight  months  to  complete. 
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starting  in  last  six  months  of  Year  1  and  two 
months  of  Year  2) . 

*  Digitize  the  400  analog  mammographic  images 
with  a  50  micron  pixel  spot  size  and  12  bits 
per  pixel  of  dynamic  range . 

*  Conduct  a  review  of  the  digitized  images  using 
the  grayscale  display  workstations  (2048  x 
2560  X  8/12  bits)  in  the  PACS  network. 

*  Archive  the  400  digitized  mammographic  arrays 
and  the  patient  data  in  a  relational  database 
(Sybase)  on  the  PACS  network. 

*  Digitize  the  400  mammographic  images  with  a 
23 -micron  spot  size  film  digitizer  (DBA, 

Fairfax,  Virginia)  and  evaluate  any 
significant  difference  between  the  50 -micron 
and  23 -micron  spot  size  digital  arrays  by  the 
following:  (a)  registering  and  subtracting 

the  RBI  region,  multiplying  the  difference 
image  by  10,  and  calculating  the  mean  square 
error;  (b)  displaying  on  the  grayscale  display 
stations  the  two  digital  arrays  (50  micron  and 
23  micron)  and  inspecting  the  displayed  images 
to  detect  any  significant  differences  (each 
display  station  has  two  monitors) . 

Vfe  have  completed  digitizing  350  of  the  400  mammography  cases. 

These  have  been  digitized  using  a  50  micron  spot  size  and  archived 
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onto  8  millimeter  DAC  tape.  These  digital  array  images  have  been 
interactively  cropped  using  a  rectangular  window  in  order  to 
minimize  the  background  of  the  digital  image. 

We  have  encountered  two  problems  in  accomplishing  Task  1,  Aim 
2  of  the  first  year.  First,  the  software  operating  on  our  16  two- 
monitor  grayscale  workstation  was  modified  by  EMED  (E-Systems , 
EMED,  San  Antonio,  TX)  when  changing  our  PACS  from  DOS  to  a  UNIX 
operating  system.  Laser  Film  Digitized  Images  lacked  the  expected 
DICOM  header  block.  It  took  two  of  our  graduate  students  four 
months  to  discover  this  problem.  Through  the  efforts  of  our 
graduate  students,  we  are  now  able  to  load  laser  film  digitization 
mammography  films  (4K  x  4K)  into  our  PACS  and  display  them  on  the 
2K  X  2.5K  grayscale  workstations . 

Second,  our  arrangement  with  DBA  (Fairfax,  Virginia)  to 
digitize  the  400  mammographic  images  at  a  23 -micron  spot  size  (CCD 
detector)  is  not  currently  possible  for  the  following  reasons:  (1) 
DBA  expects  for  us  to  pay  $5,  000  a  month  (plus  a  maintenance  charge 
of  $800 /month)  to  install  their  CCD  film  digitizer  (we  can  not 
afford  to  do  this);  (2)  the  DBA  CCD  film  digitizer  that  we  had 
discussed  is  limited  to  8  bits  of  dynamic  range;  (3)  the  use  of  50 
micron  spot  size  on  the  laser  film  digitizer  matches  the  sampling 
frequency  required  for  analog  film-screen  mammography  images  when 
scatter  is  considered  (scatter  limits  the  radiographic  information 
to  a  spatial  resolution  of  approximately  7  lines  per  millimeter) ; 
and  (4)  the  radiographic  magnification  of  approximately  2.3  to  2.4 
enables  a  spot  size  of  50  microns  to  be  equivalent,  in  the  actual 
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We  are  currently 


image  size,  to  approximately  45  microns, 
conducting  a  study  using  23  m  spot  size  with  a  set  of  mammography 
cases  (10  normal  and  10  abnormal)  to  determine  a  measure  of  the 
image  quality,  using  mean- square -values ,  to  determine  differences 
between  50  micron  and  23  micron  spot  sizes. 

TASK  1:  Aim  3 

*  Begin  initial  ROC  curve  analysis  of 
mammographic  analog  films  (8  months  to 
complete,  final  6  months  of  Year  1  and  two 
months  of  Year  2)  by  six  readers. 

We  have  completed  the  initial  ROC  analysis  of  the  readers.  We 
have  expanded  the  number  of  readers  from  six  to  eleven.  Image 
grading  is  conducted  with  the  following  gradings. 

Masses  1  (definitely  not  present);  2  (probably  not 
present);  3  (equivocal) ;  4  (Probably  present);  5 

(definitely  present)  . 

Microcalcifications  1  (definitely  not  present);  2 
(probably  not  present);  3  (equivocal)  ;  4  (probably 

present);  5  (definitely  present)  . 

Dilated  lactiferous  ducts  1;  2;  3;  4;  5. 

Focal  areas  of  asymmetry  or  architectural  distortion  1; 

2;  3;  4;  5. 

Diagnosis  of  image  1  (definitely  benign);  2  (probably 
benign);  3  (equivocal) ;  4  (probably  malignant);  5 
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(definitely  malignant) . 


Our  initial  ROC  results  are  detailed  in  Appendix 


For  year  2  the  following  Tasks  and  Aims  are  to 
accomplished. 


YEAR  2 

TASK  1:  Aim  2 

*  Complete  digitization  of  the  collection  of  the 
analog  mammographic  films  (two  months  of  Year 
2,  began  in  Year  1) . 

TASK  1:  Aim  3 

*  Complete  ROC  analysis  of  mammographic  analog 
films  (requires  two  months  of  Year  2  to 
complete  task  began  in  the  first  year) . 

TASK  1;  Aim  3 

*  Utilize  the  collected  digitized  image  data  set 
to  perform  an  ROC  curve  analysis  (requires  six 
months)  utilizing  the  2048  x  2560  x  8/12  bit 
grayscale  display  stations  in  the  University 
of  Virginia  PACS  by  six  readers. 

TASK  2:  Aim  4 


be 
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*  Implement  the  T-1  connection  between 
Northridge  facility  and  the  University  of 
Virginia  PACS  (three  months  of  Year  2) . 

*  Test  network  for  end-to-end  fidelity. 

TASK  2:  Aim  5 

*  Design,  establish,  and  test  an  end-to-end 
quality  control  program  for  validating  a 
telemammography  system. 

*  Operate  the  telemammography  system  to  collect 

data  for  evaluating  the  quality  control 

program . 

For  Years  3  and  4  the  following  Tasks  and  Aims  are  to  be 
accomplished. 

YEAR  3 

TASK  1:  Aim  3 

*  Complete  the  ROC  analysis  of  digitized 

mammographic  images  displayed  on  2048  x  2560  x 
8/12  bit  grayscale  display  stations  in  the 
University  of  Virginia  PACS. 

TASK  2:  Aim  5 

*  Implement  the  end-to-end  quality  control 

program  for  evaluation  and  analysis. 
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TASK  3:  Aim  6 


*  Implement  a  software  data  logger  program  which 
will  record  events  on  the  telemammography 
system. 

*  Implement  the  performance  evaluation  using  the 
metrics  of  response  time,  throughput, 
reliability,  and  clinical  acceptance. 

YEAR  4 

TASK  2:  Aim  5 

*  Evaluate  the  end-to-end  image  quality  control 
protocol  for  the  teleradiology  system. 

TASK  3:  Aim  6 

*  Evaluate  the  performance  evaluation  of  the 
teleradiology  system. 

*  Continue  with  utilization  of  the  teleradiology 
system  to  increase  the  statistical  power  of 
the  analysis . 


2 . 2  Database  of  Analog  Screen  Film  Mammograms 

Appendix  lA  lists  by  case  number  the  image  database 
cases  together  with  their  ground  truth. 

2 . 3  Reader  Responses 
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Appendix  IIA  shows  an  example  of  data  collection  for 
reader  responses  for  an  individual  case  in  the  data  file. 
Appendix  I IB  illustrates  an  example  of  the  reader 
responses  for  all  cases  read  to  date  by  readers  2  and  6. 

2.4  Format  for  additional  clinical  and  image  information  to 
be  input  into  data  base  (Appendix  III) 

Additional  information  as  each  case  to  be  added  to  the 
database  including  the  following:  age,  family  history, 
history  of  previous  breast  surgery.  Additional  lesion 
characteristics  are  the  following:  size,  level  of 
suspicion,  characteristics  of  mass  (shape,  margins,  anad 
density) ,  characteristics  of  calcification  (morphology, 
number) .  This  information  is  being  verified  and  will  be 
added  to  the  database  in  the  first  quarter  of  the  second 
year. 

Complete  demographic,  clinical  information  will  be 
entered  into  the  database  during  year  2  using  Mammographic 
Clinical  History  Sheet  (Appendix  IIIA) .  Mammograms  have  been 
classified  based  on  abnormalities  identified  using  ACR 
lexicon.  The  database  will  be  expanded  and  completed  by 
adding  the  mammographic  findings  using  the  ACR  MagView  Program 
(Appendix  I I IB) . 

2 . 5  ROC  Analysis 

Appendix  IV  details  the  ROC  results  of  the  readers. 
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2 . 6  Interactive  Grayscale  Workstation  Display  Protocols 

Acceptable  display  protocols  are  critical  in  using 
interactive  gray-scale  monitors.  The  acceptability  of  a 
protocol  for  displaying  mammographic  images  may  be  judged 
in  terms  of  the  rapidity  with  which  a  user  can 
accomplish  tasks.  Image  processing  and  management  steps 
impact  the  throughput  rate  of  a  display  protocol,  as  do 
the  demands  of  mammographers  for  specific  organizations 
of  images  on  the  screen.  One  possible  display  protocol 
for  a  two-monitor  workstation  might  be  defined  as 
follows.  Monitor  1  displays  a  current  exam  (craniocaudal 
(CC)  ,  left  and  right  breasts;  mediolateral  oblique  (MLO) , 
left  and  right  breasts) .  Monitor  2  meanwhile  displays 
either  previous  exams  if  available  (CC-  L&R;  MLO  -  L&R) 
or  previous  and  current  left  CC;  previous  and  current 
left  MLO;  etc.  Data  from  the  radiology  and  the  hospital 
information  systems  are  displayed.  Pre-set  window  and 
level  functions  could  aid  throughput,  as  could 
prefetching  (acquiring  the  patients  images  from  the 
archive  file  and  storing  on  the  workstation  prior  to  the 
images  being  interpreted) .  At  UVA,  previous  examinations 
archived  on  the  long-term  archive  file  could  take  up  to 
8  minutes  to  retrieve  to  the  disk  file  at  a  specific 
workstation . 

Images  will  be  loaded  onto  digital  tape  in  a  display 
format  consistent  with  the  way  in  which  they  will  be 
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reviewed  in  the  clinical  setting.  Mammograms  are 
typically  viewed  as  mirror  images,  and  if  a  lesion  is 
identified  in  one  breast  the  two  views  of  that  breast  are 
reviewed. 

Examinations  will  be  stored  on  the  digital  tape  in 
the  following  sequences: 

1.  Left  mediolateral  Oblique  (MLO) 

Right  MLO 

2.  Left  craniocaudal  (CC) 

Right  CC 

3 .  Left  MLO  and  Left  CC 

4 .  Right  MLO  and  Right  CC 

Two  significant  efforts  are  required  to  implement 
acceptable  display  protocols  for  a  digital  mammography 
gray-scale  workstation;  (a)  development  and  evaluation  of 
the  protocols;  and  (b)  hardware  implementation. 

First .  we  will  design  several  plausible  display 
protocols.  Second,  we  will  evaluate  the  protocols  by 
transferring  40  digitized  screen-film  mammography  cases 
from  the  PACS  to  an  optical  disk.  These  cases  will  be 
equally  divided  among  masses,  microcalcifications, 
architectural  distortions, a  nd  focal  asymmetries.  The 
optical  disks  will  have  the  images  preloaded  for  each  of 
the  display  protocols  to  avoid  biasing  the  protocol 
evaluation  with  the  frustration  of  the  mammographer  in 
loading  a  prescribed  sequence.  Third,  four  UVA  and  MCV 
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Display  A 


Display  B 


Display  C 


Display  D 


mammographers  will  evaluate  the  lesions  using  each  of  the 
display  protocols.  A  reader  rating  scale  will  be  used 
for  each  case  (Example,  mass:  1  =  definitely  acceptable; 
2  =  probably  acceptable;  3  =  equivocal;  4  -  probably 

unacceptable;  and  4  =  definitely  unacceptable) .  The 

order  of  each  question  will  be  randomized  as  well  as  the 
cases.  The  reader  data  will  be  analyzed  for  the  mean 
score.  The  times  of  initiation  and  completion  of  each 
study  will  be  recorded  for  calculating  the  throughput 
times.  A  preferred  display  protocol  will  be  identified 
on  the  basis  of  the  mean  score  and  a  t-test. 

The  hardware  effort  is  to  implement  the  best  display 
protocols,  as  evaluated  by  the  mammography  readers,  onto 
a  hardware  platform.  For  the  two-monitor  system,  using 
the  UVA  system  as  a  testbed,  we  will  incorporate  the 
selected  display  protocol  onto  a  DSP  board  or  a  Hewlett- 
Packard  (HP)  accelerator  board  using  toolkits  provided  by 
HP.  The  hardware  implementation  on  the  UVA  PACS  two- 
monitor  systems  is  not  yet  approved.  HP  is  just  now 
announcing  their  new  accelerator  board;  we  expect  to  have 
it  available  on-site  by  February  1996. 

We  are  currently  evaluating  a  set  of  display 
protocols . 

2 . 6  Throughput  Performance . 

Cost-benefit  analyses  for  digital  telemammography 
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lie  in  the  distant  future,  as  they  will  need  to  reflect 
currently  nonexistent  relationships  among  costs, 
availability,  efficacy,  and  quality-of -life  features .  An 
opportunity  to  analyze  initial  costs,  however,  lies  in 
the  present,  created  by  the  availability  of  the  digital 
mammography  environment  described  in  this  application. 
We  have  devised  a  cost  analysis  method  in  which,  for  any 
well-defined  system,  time  can  be  used  to  create  a 
relationship  between  the  jobs  accomplished  per  unit  of 
time  (the  throughput  rate)  and  resources  used  (costs)  to 
accomplish  those  jobs.  This  novel  strategy  should  be 
applicable  to  any  mammography  setting,  or  for  that 
matter,  to  any  clinical  setting. 

Table  1  illustrates  the  starting  point  for  the 
analysis--a  resource  allocation  table;  the  "job"  in  this 
case  is  a  conventional  mammographic  examination,  defined 
as  beginning  with  patient  registration  and  ending  with 
the  filed  report  (see  table  legend) .  The  steps  are 
listed  in  column  1.  The  estimated  time  needed  to 
complete  each  step  (T^)  is  given  in  columns  2  of  Table 
II;  Ti  is  assumed  to  be  normally  distributed,  therefore 
10  observations  of  each  step  provide  enough  sample  points 
for  estimating  the  mean  time  values.  The  resources  used 
to  accomplish  the  steps  are  shown  as  column  headings 
(refer  to  table  legend),  and  each  resource's  points  of 
contribution  are  indicated  in  the  table  by  "1."  Thus 
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Table  I 


Resource  Allocation  for  reading  Film  Mammography 
MCV  Mammography  Department 


Step 

Clerk 

Tech 

Modality 

Film 

Processor 

Film  Room 
Personnel 

Resident 

Radi** 

ologist 

Avg. 

Time 

1 

1 

0 

0 

0 

0 

0 

0 

T1 

2 

0 

0 

0 

0 

1 

0 

0 

T2 

3 

0 

1 

1 

0 

0 

0 

0 

T3 

4 

0 

1 

0 

1 

0 

0 

0 

T4 

5 

0 

1 

0 

0 

0 

0 

0 

T5 

6 

0 

1 

1 

1 

0 

0 

0 

T6 

7 

0 

0 

0 

0 

0 

1 

0 

T7 

8 

0 

0 

0 

0 

0 

1 

0 

T8 

9 

0 

0 

0 

0 

0 

1 

1 

T9 

10* 

TIO 

11 

0 

0 

0 

0 

0 

1 

0 

Til 

12 

0 

0 

0 

0 

0 

1 

0 

T12 

13 

0 

0 

0 

0 

0 

1 

0 

T13 

14 

0 

0 

0 

0 

0 

0 

1 

T14 

15 

0 

0 

0 

0 

0 

0 

1 

T15 

16 

0 

0 

0 

0 

0 

0 

1 

T16 

17 

0 

0 

0 

0 

i 

1 

0 

0 

T17 

Steps  used:  1  =  Registration;  2  =  Prior  Film  Retrieval;  3  = 
Image  Acquistion;  4  =  Film  Processed;  5  =  Quality  Assurance;  6  = 
Re-acquisition  and  Processing;  7  =  Films  Hung;  8  =  Review  of 
Clinical  Info.;  9  =  Films  Read;  10  =  Additional  views  or 
Studies*;  11  =  Write  Early  Reading;  12  =  Report  Dication;  13  = 
Notify  Clinician;  14  =  Review  Results  with  Patients;  15  = 
Comparison  with  Prior  Films;  16  =  Dictate  Addendum  to  Report;  17 
=  Filing  of  Report. 

♦Additional  Views  for  Diagnostic  Mammogram 

1  =  Resources  Used 

2  =  Resources  Not  Used 
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Table  II 


Raw  data  for  Reading  Film  Meunmoaraphv  Bottleneck  Analysis 

MCV  Mammography  Department 


T1 

12 

13 

11 

10 

10 

10 

10 

13 

15 

11 

T2* ** *** 

1.75 

T3 

10 

13 

12 

14 

11 

13 

12 

15 

6 

26 

T4 

4 

5 

3 

6 

5 

5 

5 

5 

5 

4 

T5 

3 

1 

2 

2 

1 

1 

2 

1 

1 

2 

T6 

7 

8 

7 

T7 

0.33 

0.66 

0.17 

0.83 

0.25 

0.83 

0.17 

0.25 

0.25 

0.17 

T8 

0.33 

0 . 17 

0.25 

0.083 

0.33 

0.25 

0.33 

0.17 

0.50 

.083 

T9 

0.25 

0.58 

0.75 

0.25 

0.42 

0.75 

0.33 

1.00 

1.50 

0.41 

TIO* 

* 

Til 

0.50 

0.83 

0.33 

0.17 

0.083 

0 . 083 

0.25 

0 . 75 

0.25 

0.17 

T12 

1.42 

0.75 

0.83 

1.08 

0.50 

1.83 

0.83 

2.33 

1.75 

0.83 

T13 

2 . 08 

1.5 

n 

0.42 

0.50 

1.83 

5.33 

0 . 67 

0.75 

1.5 

1.25 

0.42 

0.75 

T15 

0.58 

0.58 

0.75 

0.92 

IB 

2.25 

Tie 

2.25 

0.58 

0.66 

T17# 

0.46 

*T2  Calculation:  24  cases  retrieved  in  42  minutes,  42min/24cases 
=  1.75  min.  per  case 

**  Additional  Studies:  Ultrasound  Study;  uses  tech,  radiologist, 
and  modality  as  resources. 

Radiologist  time:  1)  30  min.  2)  8  min. 

Technologist  time:  1)  39  min.  2)  16  min. 

***T14  =  time  required  to  sign  form  letters  for  patients.  Direct 
consultation  has  not  been  defined  or  measured. 

Note:  Review  with  Residents:  two  time  trials  -  1.83  and  2.00 

minutes . 
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described,  a  mammographic  exam  becomes,  for  our  purposes, 
a  system  comprised  of  identifiable  resources  and  steps, 
and  consequently  suited  to  established  methods  for 
systems  analysis. 

One  way  to  characterize  the  operation  of  a  system  is 
by  its  throughput  rate  (measured  in  jobs  per  second)  . 
Methods  we  have  used  previously  to  analyze  PACS 
operations- -bottleneck  analysis  and  Jackson  network 
analysis  (35 , 36) - -when  applied  to  mammography  will 
generate  the  throughput  rate  for  each  resource  involved 
in  the  "job"  of  interest,  and  ultimately  for  the  job  as 
a  whole.  Cost  can  then  be  related  to  the  system's 
throughput  rate. 

All  systems  have  an  operating  region  which  is 
bounded  by  upper  and  lower  limits  on  throughput.  To 
define  the  upper  bound  on  the  throughput,  a  bottleneck 
analysis  is  performed,  in  which  Little's  law  (37)  is  used 
to  identify  the  rate -limiting  resource  for  the  case  when 
the  whole  system  is  available  for  one  job  (eg,  for  one 
examination  in  our  model) .  Little's  law  states  that  the 
mean  number  of  jobs  (e)  within  the  system  equals  the  mean 
throughput  rate  (X)  multiplied  by  the  mean  time  in  the 
system  (t)  .  Thus,  E  =  X  T.  The  mean  throughput  rate  for 
each  resource  will  increase  until  one  resource  is 
completely  utilized  (100%) ;  this  point  of  100% 
utilization  is  termed  a  "bottleneck,  "  ie,  the  upper  bound 
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beyond  which  the  system' s  throughput  rate  cannot 
increase.  To  calculate  throughputs,  one  assumes  that 
each  resource  in  turn  is  the  bottleneck.  If  the  film 
processor  in  Table  1  is  the  bottleneck,  for  example,  its 
throughput  is  given  as  Xfum processor  =  1/ (T4  +  Tg)  .  If  there 
are  two  processors  then  processor  =  2/  (T4  +  Tg)  .  The 
resource  with  the  smallest  throughput  rate  is  the  true 
bottleneck.  In  our  example,  the  bottleneck  of  the  upper 
bound  is  the  technologist  at  0.0371  jobs/second). 

The  lower  bound  of  the  throughput  describes  the 
system  when  more  than  one  demand  is  placed  on  it  (in  our 
example,  the  demand  for  one  than  one  examination) .  The 
lower  bound  throughput  is  the  number  of  simultaneous 
demands  possible  before  the  next  demand  is  forced  to  wait 
for  service .  Lower-bound  values  shift  depending  on  the 
resource  being  evaluated.  for  example,  we  may  want  to 
examine  the  bound  placed  on  our  model  mammography  unit's 
throughput  by  the  number  of  clerks  (C)  ;  in  which  case,  we 
define  T<,ierk  the  time  used  for  the  clerk's  task(s)  and 
Tsystem  the  sum  of  time  used  by  all  other  system 
resources  {Ttech  +  T„odaiity  +  ...etc).  We  then  calculate 
the  lower  bound  on  the  mean  throughput  as  c/I 

*^clerk  ^ 

fT  ) 

'  -^system'  • 

Figure  demonstrates  mean  throughput  rates  (X)  as 
a  function  of  the  number  of  examinations  in  our  model 
system,  lower-bounded  in  this  case  by  the  number  of 
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clerks.  The  area  between  the  upper  bound  (determined  by 
bottleneck  values  for  the  resources)  and  the  lower  bound 
(all  resources  are  busy  as  of  the  current  demand)  is  the 
operating  region  for  the  system.  Once  this  region  has 
been  defined,  it  becomes  possible  to  validate  its 
predicted  upper  and  lower  bounds  in  a  real  system,  which 
will  follow  the  predictions  if  the  model  has  accurately 
described  the  steps  and  resources  necessary  to  the  job. 
If  the  real  setting  behaves  differently,  the  model  can  be 
corrected  by  incorporating  the  differences  in  steps  or 
resources . 

The  costs  for  each  resource  can  be  determined  from 
financial  records.  The  UVA  Hospital  has  recently 
implemented  an  accounting  system  capable  of  generating 
detailed  cost  analyses;  for  purposes  of  this  study, 
actual  costs  will  be  supplemented  with  imputed  costs  for 
noncommercial  equipment .  One  purpose  of  the 
throughput /cost  analysis  is  to  establish  what  economists 
describe  as  a  "production  function."  This  is  a 
mathematical  relationship  between  the  mix  of  input 
resources,  the  total  volume  of  production,  and  the 
cost/unit  of  output .  In  a  typical  industrial  production 
situation,  one  observes  that  cost /unit  is  relatively  high 
when  volume  is  very  low,  but  declines  as  volume 
increases.  At  some  point  there  are  "diseconomies  of 
scale"  (Something  like  "bottlenecks")  and  cost/unit 
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begins  to  increase .  We  would  expect  in  our  analyses  to 
observe  a  cost  function  where  the  cost  per  job  first 
decreases  as  throughput  expands  and  then  at  some  point 
increases . 

3 .  Conclusions 

3.1  Implications  of  Completed  Work. 

At  the  end  of  the  first  year  of  activity, 
we  find  researches  across  the  country  asking 
if  they  could  obtain  the  400  cases  of 
digitized  screen  film  mammography  and  the 
patient  data.  We  plan  to  complete  this 
database  by  March  1996.  This  database  will  be 
sent  to  requesting  researchers  with  prior 
approval  of  the  U.S.  Army  for  the  cost  of  the 
8  mm  DAC  Tape.  All  patient  data  has  been 
appendixed  to  the  digitized  screen- film 
mammography  using  the  DICOM  3.0  Data  format 
standard.  We  envision  that  this  database  will 
be  used  by  other  researchers  for  the  following 
possible  projects. 

a.  Developing  and  evaluating  computer 
aid  diagnosis  algorithms  for  digital 
mammography.  ROC  data  on  cases 
completed  to  date  (Appendix  IV)  have 
shown  that  the  case  selection 
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reflects  adequate  range  of  subtlety. 

b.  Develop  and  evaluate  improved 
interactive  grayscale  workstation 
display  protocols . 

c.  Utilize  the  jackknife  ROC  analysis 
on  a  database  of  proven  images  and 
patient  data. 

d.  Evaluate  and  correlate  the  type  and 
subtlety  of  breast  lesions  versus 
reader  responses  on  digital  and 
analog  images. 

e.  Evaluate  image  compression  ratios 
for  ROC  metrics . 

Our  first  year  of  ROC  analysis  for  200 
normals  and  and  200  abnormals  of  digitized 
screen-films  has  reinforced  the  need  for 
implementing  other  methods  for  evaluating 
reader  responses.  ROC  analysis  is  very  time 
consuming,  often  requiring  convincing 
mammographers  to  participate  in  such  studies. 
It  is  often  suggested  that  qualified 

mammographers  can  read  200  to  300  cases  per 
day  for  ROC  studies.  The  existing  database  of 
200  normals  and  200  abnormals  will  serve  as  a 
resource  for  those  researchers  engaged  in 
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evaluating  reader  analysis. 

We  have  just  begun  our  studies  on  the  use 
of  the  ratio  of  throughput /cost  as  a  means  of 
evaluating  cost  of  telemammography. 

Throughput  (jobs/second)  is  a  production 
measure  used  in  evaluating  computer  networks. 

As  the  throughput  increases  in  a 
telemammography  system,  the  cost  will  increase 
linearly  until  a  bottleneck  is  reached.  Then 
the  throughput  can  only  increase  by  spending 
additional  resources  to  alter  the  bottlenecks . 

Studies  are  badly  needed  in  determining  the  optimum 
image  compression  ratio  for  use  in  telemammography 
systems.  Our  initial  evaluation  is  suggesting  the  use  of 
wavelet  algorithms  at  50  to  1.  Such  studies  require  ROC 
studies  to  be  accepted  by  the  Radiology  profession.  A  30 
to  1  compression  ratio  means  that  a  single  digital 
channel  (DS-0,  64K  bits/sec)  can  be  used  to  transmitt 
digital  mammography  images  at  less  than  a  minute.  Such 
a  compression  ratio  will  also  significantly  reduce  the 
amount  of  storage  media  for  long-term  storage. 

3 . 2  Recommended  Changes 

Upon  completion  of  our  first  year  of 
studying  telemammography  systems,  we  have 
several  recommended  changes . 

First,  the  jackknife  methodology  for 
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conducting  ROC  analysis  should  be  the  method 
of  choice.  This  method  involves  the  analysis 
of  variance  (ANOVA)  of  the  pseudovalues 
computed  by  the  Quencville-Tikey  version  of 
the  jackknife.  This  experimental  design 
permits  the  comparison  of  multiple  treatments, 
defined  in  our  study  as  analog  screen- film 
cases  (treatment  1)  and  grayscale  displayed 
cases  (treatment  2)  . 

Second,  the  statistical  power  obtained  is 
significant  when  we  have  200  normals  and  200 
abnormals  and  with  10  readers.  The  cases  have 
been  carefully  selectd  and  the  10  readers  are 
qualified  mammographers .  We  have  learned  that 
completion  of  this  number  of  cases  by  all 
readers  is  very  difficult  to  accomplish.  The 
readers  are  not  often  available  when  they  are 
needed.  The  number  of  cases  (400)  creates  a 
management  difficulty.  It  may  be  that  100 
normals  and  100  abnormals  would  be  better  with 
the  use  of  15  readers. 
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APPENDIX  I 


CASE  # 
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NORMAL 

199 

3 

NORMAL 

200 

3 

ABN 

MASS,  CA 

M 

201 

2 

NORMAL 

202 

1 

ABN 

M 

203 

2 

NORMAL 

204 

3 

NORMAL 

205 

2 

ABN 

M 

206 

2 

ABN 

B 

207 

3 

NORMAL 

208 

4 

ABN 

MASS 

B 

209 

3 

NORMAL 

210 

2 

ABN 

CA 

M 

211 

1 

NORMAL 

212 

4 

ABN 

M 

213 

4 

NORMAL 

214 

4 

ABN 

CA 

M 

215 

1 

NORMAL 

216 

4 

ABN 

CA 

M 

217 

2 

NORMAL 

218 

4 

NORMAL 

219 

3 

ABN 

CA 

M 

220 

1 

NORMAL 

221 

3 

NORMAL 

222 

3 

ABN 

223 

1 

NORMAL 

224 

1 

NORMAL 

225 

1 

NORMAL 

226 

1 

NORMAL 

227 

3 

NORMAL 

228 

3 

ABN 

CA 

229 

1 

NORMAL 

230 

3 

ABN 

FAD 

231 

3 

ABN 

MASS 

232 

4 

NORMAL 

233 

3 

ABN 

MASS 

234 

3 

ABN 

AD 

235 

1 

NORMAL 

236 

1 

NORMAL 

237 

4 

ABN 

CA 

238 

1 

ABN 

CA 

239 

2 

NORMAL 

240 

1 

NORMAL 

241 

2 

ABN 

242 

3 

NORMAL 

243 

1 

ABN 

MASS 

244 

2 

ABN 

MASS 

245 

3 

NORMAL 

246 

4 

ABN 

247 

4 

ABN 

CA 

248 

1 

ABN 

249 

2 

ABN 

CA 

250 

1 

ABN 

CA 

251 

3 

ABN 

MASS 

252 

3 

NORMAL 

253 

2 

ABN 

MASS 

254 

1 

ABN 

CA 

255 

4 

NORMAL 

256 

4 

NORMAL 

257 

2 

ABN 

CA 

258 

1 

NORMAL 

259 

2 

ABN 

MASS 

260 

4 

NORMAL 

261 

3 

ABN 

CA 

262 

2 

NORMAL 

263 

1 

ABN 

MASS 

264 

1 

ABN 

CA 

265 

4 

ABN 

CA 

M 


M 

B 

M 

M 

B 
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266 

2 

NORMAL 

267 

1 

NORMAL 

268 

1 

ABN 

FAD 

269 

2 

NORMAL 

270 

2 

ABN 

CA 

271 

3 

ABN 

CA 

272 

4 

NORMAL 

273 

3 

ABN 

CA 

274 

3 

NORMAL 

275 

2 

ABN 

CA 

276 

4 

ABN 

CA 

277 

2 

NORMAL 

278 

4 

NORMAL 

279 

3 

NORMAL 

280 

1 

NORMAL 

281 

3 

NORMAL 

282 

3 

ABN 

CA 

283 

4 

ABN 

CA 

284 

4 

ABN 

CA 

285 

3 

NORMAL 

286 

2 

NORMAL 

287 

1 

ABN 

MASS 

288 

4 

ABN 

CA 

289 

3 

NORMAL 

290 

1 

ABN 

MASS 

291 

2 

NORMAL 

292 

3 

NORMAL 

293 

3 

ABN 

CA 

294 

4 

NORMAL 

295 

3 

ABN 

CA 

296 

3 

ABN 

MASS 

297 

2 

NORMAL 

298 

4 

NORMAL 

299 

3 

ABN 

CA 

300 

1 

NORMAL 

301 

4 

ABN 

CA 

302 

4 

NORMAL 

303 

4 

NORMAL 

304 

3 

ABN 

CA 

305 

3 

ABN 

CA 

306 

2 

ABN 

MASS 

307 

4 

NORMAL 

308 

4 

ABN 

FAD 

309 

1 

ABN 

MASS 

310 

4 

NORMAL 

B 


M 

M 


M 

B 

B 

M 

M 
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311 

4 

ABN 

FAD 

M 

312 

2 

ABN 

MASS 

M 

313 

3 

NORMAL 

314 

3 

ABN 

CA 

M 

315 

4 

NORMAL 

316 

2 

ABN 

MASS 

M 

317 

2 

NORMAL 

318 

4 

NORMAL 

319 

3 

NORMAL 

320 

3 

ABN 

CA 

B 

321 

4 

NORMAL 

322 

2 

ABN 

CA 

M 

323 

4 

ABN 

CA 

M 

324 

4 

ABN 

CA 

B 

325 

2 

ABN 

FAD 

B 

326 

3 

NORMAL 

327 

2 

NORMAL 

328 

3 

ABN 

CA 

B 

329 

2 

NORMAL 

330 

2 

ABN 

MASS 

B 

331 

2 

NORMAL 

332 

1 

NORMAL 

333 

4 

ABN 

MASS 

M 

334 

2 

ABN 

FAD 

M 

335 

1 

ABN 

MASS 

M 

336 

2 

ABN 

MASS 

M 

337 

2 

ABN 

CA 

B 

338 

2 

NORMAL 

339 

3 

NORMAL 

340 

2 

NORMAL 

341 

3 

ABN 

CA 

M 

342 

3 

NORMAL 

343 

3 

NORMAL 

344 

2 

NORMAL 

345 

2 

NORMAL 

346 

3 

ABN 

CA 

M 

347 

2 

NORMAL 

348 

2 

ABN 

MASS 

B 

349 

2 

NORMAL 

350 

2 

NORMAL 

KEY 


PARENCHYMAL  DENSITY: 

1=FATTY 

2=SCATTERED  FIBROGLANDULAR  TISSUE 
3=HETEROGENEOUSLY  DENSE 
4=  EXTREMELY  DENSE 


GROUP: 

NORMAL=NORMAL 
ABN  =  ABNORMAL 

FINDINGS: 

MASS=MASS 

FAD  =FOCAL  ASYMMETRIC  DENSITY 
AD  =ARCHITECTURAL  DISTORTION 
CA  =CALCIFICATIONS 

DIAGNOSIS: 

M=MALIGNANT 

B=BENIGN 
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APPENDIX  II  A 


READERS  RESPONSES  TO  CASE  84 


MASS 

CALCIFICATION 

FAD/AD 

DIAGNOSIS 

READER  # 

1 

5 

1 

3 

6 

2 

5 

1 

2 

5 

2 

2 

I 

1 

4 

I 

5 

I 

4 

3 

1 

I 

2 

2 

2 

I 

2 

1 

2 

7 

I 

5 

1 

4 

10 

I 

5 

1 

3 

11 

KEY  TO  FINDINGS: 

KEY  TO  DIAGNOSIS 

I=DEFINITELY  NOT  PRESENT 

INDEFINITELY  BENIGN 

2=PROBABLY  NOT  PRESENT 

2=PROBABLY  BENIGN 

3=EQUIVOCAL  3=EQUIV0CAL 

4=PROBABLY  PRESENT  4=PROBABLY  MALIGNANT 


5=DEFINITELY  PRESENT 


5=DEFINITLEY  MALIGNANT 


APPENDIX  II  B 


READERS  SPECIFIC  RESPONSES 


READER  6 


TRUE  NORMALS 

TRUE  BENIGN 

TRUE  MALIGNANT 

(149) 

(66) 

(84) 

DEFINITELY  BENIGN 

010 

01 

00 

PROBABLY  BENIGN 

121 

17 

14 

EQUIVOCAL 

017 

41 

30 

PROBABLY  MALIGNANT 

001 

07 

25 

DEFINITELY  MALIGNANT 

000 

00 

15 

READER  2 


TRUE  NORMALS 
(149) 

TRUE  BENIGN 
(66) 

TRUE  MALIGNANT 
(84) 

DEFINITELY  BENIGN 

28 

02 

01 

PROBABLY  BENIGN 

62 

17 

06 

EQUIVOCAL 

59 

37 

41 

PROBABLY  MALIGNANT 

00 

10 

23 

DEFINITELY  MALIGNANT 

00 

00 

13 
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CLINICAL  fflSTORY  SHEET 
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MAMMOGRAPHY  CLINICAL  HISTORY  SHEET 
HISTORY  NO.:  _  DATE  OF  SERVICE: 


NAME  :  LAST _  FIRST _ .  I . 

I 

ADDRESS:  _  2lP: 


SSN: _  DATE  OF  BIRTH: _ 

HOME  PHONE  NUMBER:  ( _ ) _ - _ ^WORK  PHONE  :  {  ) 


IS  THIS  YOUR  FIRST  MAMMOGRAM?  Y  N 

IF  NO,  WHERE  WERE  YOUR  OLD  FILMS  DONE?  _ 

WHEN  WAS  YOUR  LAST  MAMMOGRAM?  _ 

HOW  OLD  WERE  YOU  WHEN  YOUR  PERIOD  STARTED?  _ 

WHAT  IS  THE  DATE  OF  YOUR  LAST  PERIOD? 

HAVE  YOU  EVER  HAD  A  HYSTERECTOMY?  Y  N 

DID  THEY  REMOVE  YOUR  OVARIES?  Y  N 

HOW  MANY  TIMES  HAVE  YOU  BEEN  PREGNANT? 

HOW  MANY  CHILDREN  DO  YOU  HAVE? 

HOW  OLD  WERE  YOU  WHEN  YOUR  FIRST  CHILD  WAS  BORN?  _ 

DO  YOU  TAKE  BIRTH  CONTROL  PILLS? 


HAVE  YOU  EVER  HAD  CANCER?  Y  N 

IF  YES,  WHAT  KIND  OF  CANCER?  _ 


HAVE  ANY  OF  YOUR  FAMILY  MEMBERS  HAD  BREAST  CANCER? 

MOTHER  SISTER _  AUNT _  GRANDMOTHER  OTHER 

GIVE  AGE  AT  DIAGNOSIS: 


DO  YOU  HAVE  BREAST  IMPLANTS? 

IF  YES,  WHAT  KIND  OF  IMPLANTS? 

Y 

N 

DO  YOU  TAKE  HORMONES?  Y 

WHAT  KIND  OF  HORMONES?  ESTROGEN  TAMOXIFIN 

N 

PROGESTERONE  OTHER 

AT  WHAT  AGE  DID  YOU  BEGIN  TAKING  HORMONES? 

HAVE  YOU  EVER  HAD  BREAST  SURGERY? 

IF  YES,  WHEN  AND  WHICH  BREAST? 

Y 

N 

WHAT  WERE  THE  RESULTS? 

HAVE  YOU  EVER  HAD  RADIATION  THERAPY? 

IF  YES,  WHICH  BREAST  AND  IN  WHAT  YEAR? 

Y 

N 

HAVE  YOU  EVER  HAD  A  BREAST  REMOVED? 

IF  YES,  WHICH  BREAST? 

Y 

N 

HAVE  YOU  FOUND  ANY  NEW  LUMPS  IN  YOUR  BREAST? 

IF  YES,  WHICH  BREAST? 

Y 

N 

HOW  LONG  HAVE  YOU  HAD  THE  LUMP? 

HAS  THE  LUMP  CHANGED? 

DO  YOU  HAVE  ANY  OTHER  NEW  BREAST  PROBLEMS? 

IF  YES,  PLEASE  DESCRIBE: 

Y 

N 

WHEN  DID  THE  PROBLEM  START? 
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TECH  ID 


CLINICAL  FINDINGS; 

SCREENING _ 

ADDITIONAL  VIEWS _ 


FOLLOW  DP  AT  SHORT  INTERVAL_ 


PRE-RADIATION  THERAPY _ 

BLOODY  DISCHARGE _ 

BREAST  IMPLANT  PROBLEM _ 

NIPPLE  ABNORMALITY _ 

LARGE  AXILLARY  LYMPH  NODES _ 

OTHER _ _ _ _ _ _ _ 

ANGLE  OF  OBLIQUITY  ON  MLO:  30  45  60 

BREAST  ULTRASOUND;  (PLEASE  CIRCLE) 

RIGHT  LEFT  BOTH 

ADDITIONAL  VIEWS;  (PLEASE  INDICATE  WHICH  BREAST  R  OR  L) 


REVIEW  OF  OUTSIDE  STUDY _ 

PROBLEMS  INDICATED; 

PALPABLE  ABNORMALITY _ 

NON  BLOODY  DISCHARGE _ 

SKIN  THICKENING  OR  RETRACTION _ 

PAIN _  CANCER  ELSEWHERE _ 


1) _ _  2) _ 

3) _ _ _  4) - 

PLEASE  CIRCLE  MACHINE  USED; 

NELSON  CLINIC  STONY  POINT 

I  II  III  IV  I  II 


VIEW 

MAS 

KVP 

+  OR  - 

KG 

MM 

R  CC 

L  CC 

• 

R  MLO 

L  MLO 

COMMENTS : 
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MAMMOGRAPHIC  FINDINGS 
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Wli|%rtew“ 


Patient  ID: 


Finding  check-off 
sheets 


Patient  Name: 


Examination  Date: 


□  Prior  study  dates  compared:  /  /  /  /  /  / 


Finding'  _ of 


□  Negative  exam 

Tissue  Density 

Recommendation 

□  Mammogram 

□  Almost  entirely  fat 

□  Normal  interval  screening 

□  Ultrasound 

□  Scattered  fibroglandular 

in _ months  or  by  age _ 

□  Ductography 

densities 

□  Any  decision  to  biopsy  should  be 

□  Heterogeneously  dense 

based  on  clinical  assessment 

□  Extremely  dense 

Initials: 

Q  Non-Negative  Finding 


□  Finding  correlates  to  clinical  exam  finding  in  QL  UR  LIB  breasifs)  at 

(location) 

G  Follow-up 

Q  Follow-up  of  prior  finding 

;  □  Change 

Q  Decrease  in  number  of 

G  FoIIow-up  of  procedure 

in  UL  Or  GB  breast(s) 

Q  No  significant  changes 

calcifications 

□  Lumpectomy 

at  _ 

Q  Increase  in  size 

□  Less  defined 

Q  Excisional  biopsy 

(location). 

□  Decrease  in  size 

Q  More  defined 

□  Mastectomy 

□  Increase  in  number  of 

Q  Completely  removed 

□  Needle  biopsy 

calcifications 

□  Partially  removed 

□  Radiation  Therapy 

Finding  Side: 

□  Left  □  Right 

U  Both  □  Multiple  similar  findings:  Approximate  number: 

Mammogram 

U  Not  seen  on  mammogram 

Mass  Shape  (choose  one) 

Calcifications 

Other  findings 

Q  Round 

□  Skin 

Q  Nipple  retraction 

Tissue  Density  (choose  one) 

□  Oval 

□  Vascular 

Q  Skin  thickening 

U  Almost  entirely  fat 

□  Lobular 

□  Coarse 

□  Trabecular  thickening 

G  Scattered  fibroglandular 

□  Irregular 

□  Large  rod-like 

□  Skin  lesion 

densities 

□  Architeaural  distortion 

□  Large  round 

□  Axilliary  adenopathy 

G  Heterogeneously  dense 

□  Tubular  density/solitary 

□  Eggshell  or  rim 

□  Skin  retraaion 

G  Extremely  dense 

dilated  dua 

Q  Milk  of  calcium 

□  Architectural  distortion 

□  Intramammary  lymph  node 

□  Dystrophic 

□  Hematoma, 

□  Asymmetric  breast  tissue 

□  Punaate 

□  Post  surgical  scar 

□  Focal  asymmetric  density 

□  Amorphous  or  indistinct 

□  Heterogeneous  or  pleomorphic  Implant  Findings 

Margins  (choose  one) 

□  Fine  and/or  branching 

□  Asymmetric  implant 

□  Circumscribed 

□  Spherical  or  lucent-centered 

□  Calcified  implant 

□  Microlobulated 

□  Suture 

□  Distorted  implant 

□  Obscured 

□  Fibrosed  implant 

□  Indistina 

Distribution  (choose  one) 

□  Herniated  implant 

□  Spiculated 

Q  Grouped  or  clustered 

□  Ruptured  implant 

Q  Segmental 

□  Free  silicone 

Density  (choose  one) 

□  Regional 

□  Capsular  coniraaion 

% 

□  High  density 

□  Linear 

□  Low  density 

□  Diffuse/scaitered 

□  Isodense 

□  Fat  containing 

53 

Ductography 

□  Not  seen  on  Duciosram 


□  Extravasation 
Q  Duct  narrowing 

□  Cyst  fill 


Q  Intraluminal  filling  defect 

□  Duct  ectasia 

□  Multiple  filling  defects 

□  Abrupt  duct  termination 


Ultrasound 

□  Not  seen  on  Ultrasound 


Modifiers 

□  Anechoic 

□  Hypoechoic 

□  Hyperechoic 

□  Isoechoic 

□  Mixed  echogenicity 

□  Posterior  acoustic  shadowing 

□  Posterior  acoustic  enhancement 


Finding 

□  Simple  Q^st 

□  Complex  Cyst 

□  Intracystic  lesion 

□  Duaeaasia 

□  Solid  mass 


Size  and  Location 


Assessment  and 
Recommendation 


Location 


.millimeters  □  Subareolar 
□  Central 

- o’clock  □  Axillary  tail 


Additional  Evaluation  Needed 

□  Cyst  aspiration 

□  Additional  projections 

□  Magnification  views 

□  Spot  compression 

□  Clinical  correlation 

□  Ultrasound  exam 


Benign 

□  Normal  interval  screening  in 
_ months 

□  Cyst  aspiration 

□  Any  decision  to  biopsy  should  be 
based  on  clinical  assessment 

Probably  Benign 

□  Short  interval  follow-up  in 
_ months 


Depth 

□  Anterior 

□  Middle 

□  Posterior 


Suspicious 

□  Biopsy  should  be  consideret 

□  Needle  localization  and  biof 

□  Histology  using  core  biopsy 

Malignant 

□  Biopsy  should  be  considered 

□  Needle  localization  and  biop 

□  Histology  using  core  biopsy 

□  Appropriate  action  should  b‘ 
taken 


APPENDIX  IV  A 


READER 

02 

03 

04 

05 

06 

07 

09 

10 

11 

12 

13 


READER  STATUS  REPORT 
CASES  READ  (  AS  OF  POOP-OS’) 

1-299 

1-299 

1-299 

1-350 

1-300 

1-300 

1-250 

1-200 

1-200 

1-100 

1-100 
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APPENDIX  IV  B 


ROC  RESULTS  OF  READERS  FOR  ANALOG  IMAGES 
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1 


R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  2,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  59. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0403  0.0537  0.4094  0.7047  1.0000 
TPF:  0.0000  0.5932  0.7627  0.8983  0.9492  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3744  B=  0.5371 

Z(K) - 0.5376  0.2287  1.6104  1.7480 

LOGL=  -271.0403 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.4609  B=  0.6171 

Z(K)=  -0.5266  0.2289  1.4998  1.9286 

LOGL=  -264.3502 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0525  0.0197  0.0050  0.0052  0.0045  0.0009 

B  0.0197  0.0173  0.0020  0.0011  -0.0052  -0.0117 

Z(  1)  0.0050  0.0020  0.0116  0.0062  0.0030  0.0021 

Z(  2)  0.0052  0.0011  0.0062  0.0105  0.0055  0.0047 

Z(  3)  0.0045  -0.0052  0.0030  0.0055  0.0234  0.0231 

Z(  4)  0.0009  -0.0117  0.0021  0.0047  0.0231  0.0397 


CORRELATION  MATRIX: 


A 

1.0000 

0.6523 

0.2014 

0.2210 

0.1284 

0.0187 

B 

0.6523 

1.0000 

0.1390 

0.0823 

-0.2592 

-0.4476 

Z( 

1) 

0.2014 

0.1390 

1.0000 

0.5645 

0.1807 

0.0962 

Z( 

2) 

0.2210 

0.0823 

0.5645 

1.0000 

0.3543 

0.2323 

z  ( 

3) 

0.1284 

-0.2592 

0.1807 

0.3543 

1.0000 

0.7586 

Z( 

4) 

0.0187 

-0.4476 

0.0962 

0.2323 

0.7586 

1.0000 

AREA  = 

0.8931 

STD. 

.  DEV. (AREA)  = 

0.0290 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


1 

44. 

3. 


2 

44. 

3. 


3 

53. 

8. 


4 

2. 

10. 


5 

6. 

35. 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.4487 

( 

0.2635 

/ 

0.6460 

) 

0.010 

0,5100 

( 

0.3324 

F 

0.6855 

) 

0.020 

0.5766 

( 

0.4124 

r 

0.7283 

) 

0.030 

0.6179 

( 

0.4638 

r 

0.7551 

) 

0.040 

0.6481 

( 

0.5019 

r 

0.7751 

) 

0.050 

0.6721 

( 

0.5322 

r 

0.7911 

) 

0.060 

0.6919 

( 

0.5573 

r 

0.8046 

) 

0.070 

0,7088 

( 

0.5787 

f 

0.8163 

) 

0.080 

0.7236 

( 

0.5973 

r 

0.8266 

) 

0.090 

0.7367 

( 

0.6137 

r 

0.8359 

) 

0.100 

0.7485 

( 

0.6284 

r 

0.8443 

) 

0.110 

0.7592 

( 

0.6416 

f 

0.8520 

) 

0.120 

0.7690 

( 

0.6537 

f 

0.8591 

) 

0.130 

0.7781 

( 

0.6647 

r 

0.8656 

) 

0.140 

0.7865 

( 

0.6749 

r 

0.8718 

) 

0.150 

0.7943 

( 

0.6844 

r 

0.8775 

) 

0.200 

0.8268 

( 

0.7234 

r 

0.9015 

) 

0.250 

0.8520 

( 

0.7531 

r 

0.9200 

) 

0.300 

0.8723 

( 

0.7771 

r 

0.9348 

) 

0.400 

0.9040 

( 

0.8147 

r 

0.9568 

) 

0.500 

0.9280 

( 

0.8442 

r 

0.9719 

) 

0.600 

0.9471 

( 

0.8693 

f 

0.9826 

) 

0.700 

0.9628 

( 

0.8921 

f 

0.9901 

) 

0.800 

0.9762 

( 

0.9142 

r 

0.9952 

) 

0.900 

0.9878 

( 

0.9384 

r 

0.9985 

) 

0.950 

0.9934 

( 

0.9538 

f 

0.9995 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0269, 

0.6067) 

(0.0102, 

0.5118) 

(0.0620, 

0.6956) 

(0.0668, 

0.7038) 

(0.0360, 

0.6369) 

(0.1151, 

0.7644) 

(0.4095, 

0.9065) 

(0.3338, 

0.8841) 

(0.4887, 

0.9256) 

< 

00 

o 

o 

o 

0.9629) 

(0.6238, 

0.9511) 

(0.7696, 

0.9723) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  2,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  ^ 


RESPONSE  DATA: 


CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

1 

127. 

12. 

2  3 

14.  5. 

6.  4. 

4 

0. 

11. 

5 

3. 

49. 

OBSERVED  OPERATING  POINTS: 
FPF:  0.0000  0.0201  0.0201 

0.0537 

0.1477  1.0000 
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ABNORMAL IT' 
=  82. 


TPF:  0.0000  0.5976  0.7317  0.7805  0.8537  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.8080  B=  0.6820 

Z(K)=  1.0466  1.6104  1.9514  2.0514 

LOGL=  -191.9226 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.8169  B=  0.6938 

Z(K)=  1.0526  1.5531  1.8560  2.2459 

LOGL=  -184.4330 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


'variance-covariance  MATRIX: 


A 

0.1216 

0.0618 

0.0150 

0.0032 

-0.0095 

-0.0326 

B 

0.0618 

0.0410 

0.0038 

-0.0066 

-0.0169 

-0.0348 

Z( 

1) 

0.0150 

0.0038 

0.0159 

0.0128 

0.0112 

0.0092 

Z( 

2) 

0.0032 

-0.0066 

0.0128 

0.0233 

0.0235 

0.0254 

Z( 

3) 

-0.0095 

-0.0169 

0.0112 

0.0235 

0.0350 

0.0406 

Z( 

4) 

-0.0326 

-0.0348 

0.0092 

0.0254 

0.0406 

0.0667 

CORRELATION  MATRIX: 

A 

1.0000  0.8756 

0.3411 

0.0600 

-0.1452 

-0.3622 

B 

0.8756  1.0000 

0.1502  • 

-0.2124 

-0.4452 

-0.6656 

Z( 

1) 

0.3411  0.1502 

1.0000 

0.6645 

0.4752 

0.2833 

Z( 

2) 

0.0600  -0.2124 

0.6645 

1.0000 

0.8240 

0.6448 

Z( 

3) 

-0.1452  -0.4452 

0.4752 

0.8240 

1.0000 

0.8409 

Z( 

4) 

-0.3622  -0.6656 

0.2833 

0.6448 

0.8409 

1.0000 

AREA  =  0.9323 

STD 

.  DEV.  (AREA)  = 

0.0231 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.5118 

( 

0.3057 

r 

0.7147 

) 

0.010 

0.5803 

( 

0.3971 

f 

0.7473 

) 

0.020 

0.6524 

( 

0.4987 

f 

0.7843 

) 

0.030 

0.6956 

( 

0.5595 

f 

0.8089 

) 

0.040 

0.7264 

( 

0.6018 

r 

0.8279 

) 

0.050 

0.7503 

( 

0.6334 

r 

0.8438 

) 

0.060 

0.7697 

( 

0.6581 

f 

0.8574 

) 

0.070 

0.7861 

( 

0.6781 

f 

0.8693 

) 

0.080 

0.8001 

( 

0.6947 

r 

0.8799 

) 

0.090 

0.8123 

( 

0.7088 

r 

0.8894 

) 

0.100 

0.8232 

( 

0.7209 

r 

0.8979 

) 

0.110 

0.8329 

( 

0.7315 

f 

0.9056 

) 

0.120 

0.8417 

( 

0.7409 

f 

0.9126 

) 

0.130 

0.8498 

( 

0.7493 

r 

0.9190 

) 

0.140 

0.8571 

( 

0.7569 

r 

0.9248 

) 

0.150 

0.8639 

( 

0.7638 

¥ 

0.9302 

) 

0.200 

0.8912 

( 

0.7911 

f 

0.9511 

) 

0.250 

0.9114 

( 

0.8111 

f 

0.9654 

) 

0.300 

0.9269 

( 

0.8269 

¥ 

0.9753 

) 
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0.400 

0.9496 

( 

0.8517 

/ 

0.9874 

) 

0.500 

0.9654 

( 

0.8715 

f 

0.9938 

) 

0.600 

0.9768 

( 

0.8887 

r 

0.9972 

) 

0.700 

0.9854 

( 

0.9047 

r 

0.9989 

) 

0.800 

0.9918 

( 

0.9210 

r 

0.9997 

) 

0.900 

0.9966 

( 

0.9395 

r 

0.9999 

) 

0.950 

0.9985 

( 

0.9519 

f 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0124, 

0.6021) 

(0.0030, 

0.4632) 

(0.0410, 

0.7291) 

(0.0317, 

0.7017) 

(0.0131, 

0.6083) 

(0.0682, 

0.7834) 

(0.0602, 

0.7702) 

(0.0320, 

0.7026) 

(0.1049, 

0.8282) 

(0.1463, 

0.8614) 

(0.0968, 

0.8199) 

(0.2103, 

0.8958) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  2,  FAS /AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  93.  27. 

CASES  6 .  0 . 


3  4  5 

10.  10.  9. 

2.  8.  11. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0604  0.1275  0.1946  0.3758  1.0000 
TPF:  0.0000  0.4074  0.7037  0.7778  0.7778  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3079  B=  0.8539 

Z(K)=  0.3160  0.8608  1.1383  1.5517 

LOGL=  -209.1670 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.1742  B=  0.8234 

Z(K)=  0.3269  0.8058  1.0795  1.6264 

LOGL=  -208.0945 

chi-square  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ; 


VARIANCE-COVARIANCE  MATRIX : 

A  0.1161  0.0609  0.0099  0.0079  0.0057  -0.0022 
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B 

0.0609 

0.0596 

0.0035 

-0.0003 

-0.0037 

-0.0152 

Z( 

1) 

0.0099 

0.0035 

0.0109 

0.0082 

0.0071 

0.0053 

Z( 

2) 

0.0079 

-0.0003 

0.0082 

0.0129 

0.0114 

0.0095 

Z( 

3) 

0.0057 

-0.0037 

0.0071 

0.0114 

0.0155 

0.0134 

Z( 

4) 

-0.0022 

-0.0152 

0.0053 

0.0095 

0.0134 

0.0283 

A 

CORRELATION  MATRIX: 
1.0000  0.7313  0.2783 

0.2036 

0.1354 

-0.0388 

B 

0.7313  1.0000 

0.1356  - 

-0.0097 

-0.1210 

-0.3698 

Z( 

1) 

0.2783  0.1356 

1.0000 

0.6919 

0.5454 

0.3029 

Z( 

2) 

0.2036  -0.0097 

0.6919 

1.0000 

0.8055 

0.4983 

Z( 

3) 

0.1354  -0.1210 

0.5454 

0.8055 

1.0000 

0.6430 

Z( 

4) 

-0.0388  -0.3698 

0.3029 

0.4983 

0.6430 

1.0000 

AREA  =  0.8177 

STD. 

.  DEV. (AREA)  - 

0.0524 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1718 

( 

0.0344 

f 

0.4705 

) 

0.010 

0.2291 

( 

0.0648 

f 

0.5128 

) 

0.020 

0.3025 

( 

0.1171 

r 

0.5617 

) 

0.030 

0.3539 

( 

0.1613 

r 

0.5946 

) 

0.040 

0.3945 

( 

0.1999 

f 

0.6205 

) 

0.050 

0.4284 

( 

0.2341 

r 

0.6422 

) 

0.060 

0.4577 

( 

0.2647 

r 

0.6614 

) 

0.070 

0.4836 

( 

0.2924 

r 

0.6786 

) 

0.080 

0.5068 

( 

0.3176 

r 

0.6944 

) 

0.090 

0.5279 

( 

0.3407 

f 

0.7091 

) 

0.100 

0.5473 

( 

0.3618 

r 

0.7228 

) 

0.110 

0.5652 

( 

0.3813 

f 

0.7357 

) 

0.120 

0.5819 

( 

0.3994 

r 

0.7480 

) 

0.130 

0.5974 

( 

0.4162 

r 

0.7596 

) 

0.140 

0.6120 

( 

0.4318 

f 

0.7707 

) 

0.150 

0.6258 

( 

0.4464 

f 

0.7812 

) 

0.200 

0.6849 

( 

0.5071 

r 

0.8276 

) 

0.250 

0.7321 

( 

0.5534 

r 

0.8652 

) 

0.300 

0.7712 

( 

0.5905 

f 

0.8955 

) 

0.400 

0.8330 

( 

0.6483 

r 

0.9396 

) 

0.500 

0.8798 

( 

0.6937 

/ 

0.9673 

) 

0.600 

0.9166 

( 

0.7329 

r 

0.9840 

) 

0.700 

0.9458 

( 

0.7695 

r 

0.9933 

) 

0.800 

0.9691 

( 

0.8068 

r 

0.9979 

) 

0.900 

0.9871 

( 

0.8502 

r 

0.9997 

) 

0.950 

0.9943 

( 

0.8798 

r 

0.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0519,  0.4345) 


(0.0252,  0.3313)  (0.0974,  0.5424) 
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(0.1402,  0.6123)  (0.0929,  0.5337)  (0.2016,  0.6865) 

(0.2102,  0.6952)  (0.1519,  0.6284)  (0.2798,  0.7561) 

(0.3719,  0.8173)  (0.2974,  0.7692)  (0.4515,  0.8586) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  2,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.3960  0.8121  1.0000 
TPF:  0.0000  0.0867  0.3067  0.8267  0.9800  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2603  B=  0.8325 

Z(K)=  -0.8855  0.2634  2.6112  2.7112 

L0GL=  -397.9249 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ; 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.2232  B=  0.6858 

Z(K)=  -0.9217  0.3090  2.5500  3.7878 

LOGL=  -345.4767 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0189  0.0045  0.0052  0.0060  0.0056  -0.0005 
B  0.0045  0.0099  0.0027  -0.0001  -0.0269  -0.0459 
Z(  1)  0.0052  0.0027  0.0140  0.0051  -0.0023  -0.0073 
Z(  2)  0.0060  -0.0001  0.0051  0.0103  0.0087  0.0091 
Z(  3)  0.0056  -0.0269  -0.0023  0.0087  0.1115  0.1582 
Z(  4)  -0.0005  -0.0459-0.0073  0.0091  0.1582  0.2801 


CORRELATION  MATRIX: 

A  1.0000  0.3270  0.3178  0.4283  0.1216  -0.0072 
B  0.3270  1.0000  0.2319  -0.0120  -0.8074  -0.8706 
Z(  1)  0.3178  0.2319  1.0000  0.4245  -0.0585  -0.1170 
Z(  2)  0.4283  -0.0120  0.4245  1.0000  0.2583  0.1695 
Z(  3)  0.1216  -0.8074  -0.0585  0.2583  1.0000  0.8953 
Z(  4)  -0.0072  -0.8706  -0.1170  0.1695  0.8953  1.0000 


AREA  =  0.8435  STD.  DEV. (AREA)  =  0.0258 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
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1 

28. 

3. 


2 

62. 

23. 


3 

59. 

78. 


4 

0. 

33. 


5 

0. 

13. 


FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.2934 

( 

0.1514 

t 

0.4775 

) 

0.010 

0.3548 

( 

0.2065 

r 

0.5294 

) 

0.020 

0.4264 

( 

0.2779 

r 

0.5864 

) 

0.030 

0.4734 

( 

0.3282 

r 

0.6222 

) 

0.040 

0.5089 

( 

0.3680 

r 

0.6488 

) 

0.050 

0.5379 

( 

0.4012 

f 

0.6702 

) 

0.060 

0.5623 

( 

0.4298 

r 

0.6881 

) 

0.070 

0.5836 

( 

0.4550 

r 

0.7037 

) 

0.080 

0.6024 

( 

0.4776 

r 

0.7174 

) 

0.090 

0.6193 

( 

0.4981 

r 

0.7297 

) 

0.100 

0.6347 

( 

0.5169 

r 

0.7409 

) 

0.110 

0.6488 

( 

0.5341 

r 

0.7512 

) 

0.120 

0.6618 

( 

0.5502 

r 

0.7608 

) 

0.130 

0.6739 

( 

0.5651 

f 

0.7696 

) 

0.140 

0.6852 

( 

0.5791 

r 

0.7779 

) 

0.150 

0.6958 

( 

0.5922 

f 

0.7857 

) 

0.200 

0.7409 

( 

0.6482 

f 

0.8191 

) 

0.250 

0.7766 

( 

0.6924 

r 

0.8460 

) 

0.300 

0.8062 

( 

0.7287 

r 

0.8684 

) 

0.400 

0,8531 

( 

0.7859 

r 

0.9045 

) 

0.500 

0.8894 

( 

0.8299 

r 

0.9322 

) 

0.600 

0.9187 

( 

0.8659 

r 

0.9541 

) 

0.700 

0.9432 

( 

0.8970 

f 

0.9713 

) 

0.800 

0.9641 

( 

0.9253 

f 

0.9846 

) 

0.900 

0.9822 

( 

0.9534 

r 

0.9942 

) 

0.950 

0.9906 

( 

0.9692 

f 

0.9977 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0001, 

0.0847) 

(0.0000, 

0.0185) 

(0.0030, 

0.2537) 

(0.0054, 

0.2996) 

(0.0007, 

0.1650) 

(0.0290, 

0.4695) 

(0.3787, 

0.8441) 

(0.3059, 

0.8093) 

(0.4560, 

0.8744) 

(0.8217, 

0.9682) 

(0.7548, 

0.9551) 

(0.8757, 

0.9780) 
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R  0  C  F  I  T  (JUNE  1993  VERSION) 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  3^  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E,G.,  THAT  ABNORMALITY 


NO.  OF  ACTUALLY  NEGATIVE  CASES  ==  149, 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  59. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  100.  41. 

CASES  6.  6. 


3  4  5 

1.  6.  1. 

3.  10.  34. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0067  0.0470  0.0537  0.3289  1.0000 
TPF:  0.0000  0.5763  0.7458  0.7966  0.8983  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.5514  B=  0.5243 

Z(K)=  0.4426  1.6104  1.6752  2.4728 

LOGL=  -197.8084 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A-  1.5534  B=  0.5279 

Z(K)=  0.4474  1.5513  1.7104  2.5634 

LOGL=  -196.4182 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.0741 

0.0289 

0.0066 

0.0035 

0.0017 

-0.0209 

B 

0.0289 

0.0197 

0.0019 

-0.0042 

-0.0064 

-0.0298 

Z( 

1) 

0.0066 

0.0019 

0.0113 

0.0065 

0.0060 

0.0035 

Z( 

2) 

0.0035 

-0.0042 

0.0065 

0.0246 

0.0240 

0.0263 

Z( 

3) 

0.0017 

-0.0064 

0.0060 

0.0240 

0.0296 

0.0336 

Z( 

4) 

-0.0209 

-0.0298 

0.0035 

0.0263 

0.0336 

0.1096 

A 

CORRELATION  MATRIX: 
1.0000  0.7573  0.2287 

0.0831 

0.0367 

-0.2316 

B 

0.7573  1.0000 

0.1262 

-0.1906 

-0.2655 

-0.6429 

Z( 

1) 

0.2287  0.1262 

1.0000 

0.3888 

0.3294 

0.0989 

Z( 

2) 

0.0831  -0.1906 

0.3888 

1.0000 

0.8887 

0.5059 

Z( 

3) 

0.0367  -0.2655 

0.3294 

0.8887 

1.0000 

0.5900 

Z( 

4) 

-0.2316  -0.6429 

0.0989 

0.5059 

0.5900 

1.0000 

AREA  =  0.9152 

STD 

.  DEV. (AREA)  = 

0.0292 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.5767 

( 

0,3939 

r 

0.7440 

) 

0.010 

0.6274 

( 

0.4620 

r 

0.7721 

) 

0.020 

0.6805 

( 

0.5340 

r 

0.8031 

) 

0.030 

0.7124 

( 

0.5769 

r 

0.8229 

) 

0.040 

0.7353 

( 

0.6072 

f 

0.8379 

) 

0.050 

0.7533 

( 

0.6305 

f 

0.8500 

) 

0.060 

0.7681 

( 

0.6492 

r 

0.8603 

) 

0.070 

0.7806 

( 

0.6649 

r 

0.8692 

) 

0.080 

0.7915 

( 

0.6782 

r 

0.8770 

) 

0.090 

0.8011 

( 

0.6898 

r 

0.8841 

) 

0.100 

0.8097 

( 

0.7001 

r 

0.8905 

) 

0.110 

0.8175 

( 

0.7092 

t 

0.8963 

) 

0.120 

0.8246 

( 

0.7175 

r 

0.9016 

) 

0.130 

0.8312 

( 

0.7250 

r 

0.9065 

) 

0.140 

0.8372 

( 

0.7319 

r 

0.9111 

) 

0.150 

0.8429 

( 

0.7383 

r 

0.9153 

) 

0.200 

0.8663 

( 

0.7643 

f 

0.9329 

) 

0.250 

0.8844 

( 

0.7841 

r 

0.9462 

) 

0.300 

0.8992 

( 

0.8001 

r 

0.9565 

) 

0.400 

0.9222 

( 

0.8256 

r 

0.9715 

) 

0.500 

0.9398 

( 

0.8461 

f 

0.9816 

) 

0.600 

0.9542 

( 

0.8641 

r 

0.9885 

) 

0.700 

0,9664 

( 

0.8811 

f 

0.9934 

) 

0.800 

0.9771 

( 

0.8985 

f 

0.9968 

) 

0.900 

0.9871 

( 

0.9188 

f 

0.9989 

) 

0.950 

0.9923 

( 

0.9329 

r 

0.9996 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


1 


(0.0052,  0.5793) 
(0.0436,  0.7423) 
(0.0604,  0.7687) 
(0.3273,  0.9061) 


(0.0007,  0.4434) 
(0.0203,  0.6817) 
(0.0315,  0.7164) 
(0.2559,  0.8863) 
R  0  C  F  I  T  (JUNE 


(0.0278,  0.7063) 
(0.0848,  0.7963) 
(0.1068,  0.8151) 
(0.4056,  0.9233) 
VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  3,  MicroCalcif ications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0268  0.0268  0.0268  0.0940  1.0000 


1 

135. 

29. 


2 

10. 

1. 


3 

0. 

1. 


4  5 

0.  4. 

5.  46. 
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TPF:  0.0000  0.5610  0.6220  0.6341  0.6463  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  0.7642  B=  0.2756 

Z(K)=  1.3170  1.7297  1.8297  1.9297 

LOGL=  -142.8031 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.7916  B=  0.3089 

Z(K)=  1.3225  1.7410  1.7930  2.0498 

LOGL=  -140.5911 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.0733  0.0328 
0.0328  0.0204 
0.0093  0.0023 
0.0032  -0.0066 
0.0052  -0.0080 
0.0172  -0.0164 


0.0093  -0.0032 
0.0023  -0.0066 
0.0204  0.0171 
0.0171  0.0314 
0.0168  0.0315 
0.0150  0.0327 


-0.0052 

-0.0080 

0.0168 

0.0315 

0.0340 

0.0357 


-0.0172 

-0.0164 

0.0150 

0.0327 

0.0357 

0.0536 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.8490 
0.8490  1.0000 
0.2413  0.1123 
0.0675  -0.2591 
0.1051  -0.3030 
0.2742  -0.4970 


0.2413  -0.0675 
0.1123  -0.2591 
1.0000  0.6764 
0.6764  1.0000 
0.6365  0.9640 
0.4546  0.7976 


-0.1051 

-0.3030 

0.6365 

0.9640 

1.0000 

0.8360 


-0.2742 

-0.4970 

0.4546 

0.7976 

0.8360 

1.0000 


AREA  =  0.7753  STD.  DEV. (AREA)  =  0.0699 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0,005 

0.4983 

( 

0.3469 

/ 

0.6499 

) 

0,010 

0.5290 

( 

0.3929 

r 

0.6618 

) 

0,020 

0.5624 

( 

0.4405 

f 

0.6785 

) 

0.030 

0.5833 

( 

0.4681 

r 

0.6918 

) 

0.040 

0,5989 

( 

0.4870 

r 

0.7032 

) 

0.050 

0.6115 

( 

0.5011 

/ 

0.7136 

) 

0.060 

0.6222 

( 

0.5121 

r 

0.7231 

) 

0.070 

0.6314 

( 

0.5210 

r 

0.7319 

) 

0.080 

0.6396 

( 

0.5284 

f 

0.7401 

) 

0.090 

0.6470 

( 

0.5347 

t 

0.7478 

) 

0.100 

0.6538 

( 

0.5400 

f 

0.7551 

) 

0.110 

0.6601 

( 

0.5447 

r 

0.7620 

) 

0.120 

0.6659 

( 

0.5489 

r 

0.7686 

) 

0.130 

0.6713 

( 

0.5526 

r 

0.7749 

) 

0.140 

0.6765 

( 

0.5559 

r 

0.7808 

) 

0.150 

0.6813 

( 

0.5589 

9 

0.7865 

) 

0.200 

0.7025 

( 

0.5707 

9 

0.8119 

) 

0.250 

0.7202 

( 

0.5791 

9 

0.8332 

) 

0.300 

0.7356 

( 

0.5856 

9 

0.8516 

) 
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0.400 

0.7622 

( 

0.5954 

/ 

0.8821 

) 

0.500 

0.7857 

( 

0.6029 

r 

0.9070 

) 

0.600 

0.8078 

( 

0.6094 

r 

0.9281 

) 

0.700 

0.8298 

( 

0.6155 

r 

0.9466 

) 

0.800 

0.8535 

( 

0.6219 

r 

0.9635 

) 

0.900 

0.8825 

( 

0.6298 

t 

0.9795 

) 

0.950 

0.9032 

( 

0.6357 

r 

0.9879 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(0.0202,  0.5629)  (0.0062,  0.5073)  (0.0552,  0.6173) 

(0.0365,  0.5939)  (0.0156,  0.5501)  (0.0761,  0.6366) 

(0.0408,  0.6001)  (0.0184,  0.5582)  (0.0817,  0.6410) 

(0.0930,  0.6491)  (0.0545,  0.6165)  (0.1487,  0.6807) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  3,  FAS/AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT’ 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0940  0.2349  0.3289  1.0000 
TPF:  0.0000  0.1481  0.5556  0.5926  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.0544  B=  1.5447 

Z(K)-  0.4426  0.7225  1.3170 

LOGL=  -180.4581 

chi-square  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.9177  B=  1.3982 

Z(K)=  0.4487  0.6788  1.3417 

LOGL=  -179.8747 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX: 

A  0.1578  0.1213  0.0189  0.0149  -0.0004 


1  2  3  4,5 

100.  14.  0.  21.  14. 

11.  1.  0.  11.  4. 
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B 

0.1213 

0.1589 

0.0070 

0.0008 

-0.0213 

Z( 

1) 

0.0189 

0.0070 

0.0113 

0.0099 

0.0069 

Z( 

2) 

0.0149 

0.0008 

0.0099 

0.0119 

0.0092 

Z( 

3) 

-0.0004 

-0.0213 

0.0069 

0.0092 

0.0207 

CORRELATION  MATRIX; 


A 

1.0000 

0.7660 

0.4469 

0.3448 

-0.0065 

B 

0.7660 

1.0000 

0.1651 

0.0182 

-0.3724 

Z( 

1) 

0.4469 

0.1651 

1.0000 

0.8521 

0.4504 

Z( 

2) 

0.3448 

0.0182 

0.8521 

1.0000 

0.5831 

Z  ( 

3) 

-0.0065 

-0.3724 

0.4504 

0.5831 

1.0000 

AREA  =  0.7033  STD.  DEV. (AREA)  =  0.0578 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION; 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.0036 

( 

0.0000 

r 

0.1186 

) 

0.010 

0.0098 

( 

0.0001 

r 

0.1550 

) 

0.020 

0.0253 

( 

0.0010 

f 

0.2037 

) 

0.030 

0.0434 

( 

0.0033 

r 

0.2402 

) 

0.040 

0.0629 

( 

0.0071 

r 

0.2707 

) 

0.050 

0.0834 

( 

0.0127 

r 

0.2978 

) 

0.060 

0.1044 

( 

0.0201 

r 

0.3225 

) 

0.070 

0.1259 

( 

0.0290 

f 

0.3456 

) 

0.080 

0.1475 

( 

0.0395 

t 

0.3675 

) 

0.090 

0.1692 

( 

0,0514 

r 

0.3885 

) 

0.100 

0.1909 

( 

0.0645 

r 

0.4089 

) 

0.110 

0.2126 

( 

0.0785 

f 

0.4287 

) 

0.120 

0.2341 

( 

0.0933 

r 

0.4482 

) 

0.130 

0.2555 

( 

0.1088 

r 

0.4673 

) 

0.140 

0.2766 

( 

0.1248 

r 

0.4863 

) 

0.150 

0.2975 

( 

0.1410 

r 

0.5051 

) 

0.200 

0.3979 

( 

0.2227 

r 

0.5969 

) 

0.250 

0.4900 

( 

0.2980 

f 

0.6845 

) 

0.300 

0.5734 

( 

0.3635 

r 

0.7640 

) 

0.400 

0.7136 

( 

0.4698 

f 

0.8857 

) 

0.500 

0.8206 

( 

0.5554 

r 

0.9551 

) 

0.600 

0.8982 

( 

0.6304 

r 

0.9864 

) 

0.700 

0.9506 

( 

0.7009 

f 

0.9972 

) 

0.800 

0.9819 

( 

0.7716 

f 

0.9997 

) 

0.900 

0.9966 

( 

0.8498 

r 

1.0000 

) 

0.950 

0.9994 

( 

0.8981 

r 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS; 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0898, 

0.1689) 

(0.0522, 

0.0881) 

(0.1446, 

0.2863) 

(0.2486, 

0.4875) 

(0.1860, 

0.3705) 

(0.3210, 

0.6056) 

(0.3268, 

0.6142) 

(0.2554, 

0.4994) 

(0.4051, 

0.7197) 
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R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
.FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  3,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT’ 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0336  0.0940  0.3758  1.0000 
TPF:  0.0000  0.1333  0.3133  0.5600  0.8400  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2702  B=  0.8959 

Z(K)=  0.3160  1.3170  1.8313  2.7112 

LOGL=  -378.3870 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.2787  B=  0.8858 

Z(K)=  0.3172  1.2914  1.9619  2.7109 

LOGL=  -376.3607 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


VARIANCE-COVARIANCE  MATRIX : 

A  0.0312  0.0155  0.0106  0.0052  -0.0025  -0.0137 
B  0.0155  0.0171  0.0034  -0.0060  -0.0167  -0.0309 
Z(  1)  0.0106  0.0034  0.0109  0.0067  0.0044  0.0016 
Z(  2)  0.0052  -0.0060  0.0067  0.0165  0.0189  0.0234 
Z(  3)  -0.0025  -0.0167  0.0044  0.0189  0.0361  0.0480 
Z(  4)  -0.0137  -0.0309  0.0016  0.0234  0.0480  0.0866 


CORRELATION  MATRIX: 

A  1.0000  0.6727  0.5739  0.2270  -0.0744  -0.2638 

B  0.6727  1.0000  0.2506  -0.3583  -0.6697  -0.8032 

Z(  1)  0.5739  0.2506  1.0000  0.4988  0.2195  0.0519 

Z(  2)  0.2270  -0.3583  0.4988  1.0000  0.7734  0.6186 

Z(  3)  -0.0744  -0.6697  0.2195  0.7734  1.0000  0.8585 

Z(  4)  -0.2638  -0.8032  0.0519  0.6186  0.8585  1.0000 


AREA  =  0.8308  STD.  DEV. (AREA)  =  0.0256 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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1 

93. 

24. 


2 

42. 

42. 


3 

9. 

37. 


4 

5. 

27. 


5 

0. 

20. 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1578 

( 

0.0665 

r 

0.3070 

) 

0.010 

0.2170 

( 

0.1098 

r 

0.3680 

) 

0.020 

0.2943 

( 

0.1759 

r 

0.4400 

) 

0.030 

0.3491 

( 

0.2280 

r 

0.4880 

) 

0.040 

0.3926 

( 

0.2717 

r 

0.5251 

) 

0.050 

0.4291 

( 

0.3095 

r 

0.5556 

) 

0.060 

0.4606 

( 

0.3431 

r 

0.5818 

) 

0.070 

0.4885 

( 

0.3731 

r 

0.6048 

) 

0.080 

0.5135 

( 

0.4004 

f 

0.6255 

) 

0.090 

0.5362 

( 

0.4253 

r 

0.6443 

) 

0.100 

0.5570 

( 

0.4482 

t 

0.6616 

) 

0.110 

0.5761 

( 

0.4694 

f 

0.6115 

) 

0.120 

0.5940 

( 

0.4891 

f 

0.6924 

) 

0.130 

0.6106 

( 

0.5075 

r 

0.7064 

) 

0.140 

0.6261 

( 

0.5247 

r 

0.7195 

) 

0.150 

0.6408 

( 

0.5408 

r 

0.7319 

) 

0.200 

0.7031 

( 

0.6090 

r 

0.7852 

) 

0.250 

0.7522 

( 

0.6622 

f 

0.8275 

) 

0.300 

0.7923 

( 

0.7055 

f 

0.8619 

) 

0.400 

0.8542 

( 

0.7729 

f 

0.9132 

) 

0.500 

0.8995 

( 

0.8245 

r 

0.9479 

) 

0.600 

0.9335 

( 

0.8665 

f 

0.9710 

) 

0.700 

0.9593 

( 

0.9024 

r 

0.9858 

) 

0.800 

0.9785 

( 

0.9342 

f 

0.9945 

) 

0.900 

0.9921 

( 

0.9638 

f 

0.9988 

) 

0.950 

0.9969 

( 

0.9790 

r 

0.9997 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0034, 

0.1308) 

(0.0005, 

0.0512) 

(0.0164, 

0.2704) 

(0.0249, 

0.3230) 

(0.0098, 

0.2150) 

(0.0560, 

0.4486) 

(0.0983, 

0.5536) 

(0.0613, 

0.4647) 

(0.1493, 

0.6398) 

(0.3755, 

0.8408) 

(0.3010, 

0.7929) 

(0.4551, 

0.8807) 
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R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIH 
OF  A  BINORMA 
FROM  RATI 


OOD  ESTIMATION 
L  ROC  CURVE 
N  G  DATA 


DATA  DESCRIPTION:  Reader  4,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  59. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  38.  61. 

CASES  2 .  3 . 


3  4  5 

40.  8.  2. 

16.  6.  32. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0134  0.0671  0.3356  0.7450  1.0000 
TPF:  0.0000  0.5424  0.6441  0.9153  0.9661  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.4749  B=  0.6394 

Z(K)=  -0.6584  0.4241  1.4979  2.2142 

LOGL=  -263.4115 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.4939  B=  0.6633 

Z(K)=  -0.6526  0.4028  1.5744  2.1027 

LOGL=  -262.5013 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ; 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.0536 

0.0202 

0.0051 

0.0058 

0.0046 

-0.0010 

B 

0.0202 

0.0175 

0.0020 

0.0010 

-0.0061 

-0.0151 

Z( 

1) 

0.0051 

0.0020 

0.0122 

0.0052 

0.0026 

0.0015 

Z( 

2) 

0.0058 

0.0010 

0.0052 

0.0108 

0.0064 

0.0055 

Z( 

3) 

0.0046 

-0.0061 

0.0026 

0.0064 

0.0251 

0.0257 

Z( 

4) 

-0.0010 

-0.0151 

0.0015 

0.0055 

0.0257 

0.0492 

CORRELATION  MATRIX: 

A  1.0000  0.6613  0.1977 

B  0.6613  1.0000  0.1374 

Z(  1)  0.1977  0.1374  1.0000 

Z(  2)  0.2416  0.0691  0.4489 

Z(  3)  0.1252  -0.2914  0.1491 

Z(  4)  -0.0192  -0.5166  0.0602 


0.2416 

0.0691 

0.4489 

1.0000 

0.3873 

0.2376 


0.1252 

-0.2914 

0.1491 

0.3873 

1.0000 

0.7321 


-0.0192 

-0.5166 

0.0602 

0.2376 

0.7321 

1.0000 


AREA  =  0.8934  STD.  DEV. (AREA)  =  0.0282 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.4149 

( 

0.2371 

r 

0.6125 

) 

0.010 

0.4803 

( 

0.3068 

r 

0.6577 

) 

0.020 

0.5523 

( 

0.3895 

r 

0.7066 

) 

0.030 

0.5972 

( 

0.4434 

r 

0.7371 

) 

0.040 

0.6302 

( 

0.4837 

r 

0.7598 

) 

0.050 

0.6564 

( 

0.5159 

r 

0.7780 

) 

0.060 

0.6781 

( 

0.5426 

r 

0.7933 

) 

0.070 

0.6967 

( 

0.5654 

r 

0.8065 

) 

0.080 

0.7129 

( 

0.5853 

f 

0.8181 

) 

0.090 

0.7272 

( 

0.6029 

r 

0.8284 

) 

0,100 

0.7401 

( 

0.6187 

r 

0.8378 

) 

0.110 

0.7518 

( 

0.6329 

f 

0.8464 

) 

0.120 

0.7625 

( 

0.6459 

r 

0.8542 

) 

0.130 

0.7724 

( 

0.6578 

t 

0.8615 

) 

0.140 

0.7815 

( 

0.6688 

/ 

0.8683 

) 

0.150 

0.7900 

( 

0.6789 

f 

0.8746 

) 

0.200 

0.8253 

( 

0.7210 

r 

0.9007 

) 

0.250 

0.8524 

( 

0.7531 

r 

0.9206 

) 

0.300 

0.8742 

( 

0.7789 

r 

0.9363 

) 

0.400 

0.9076 

( 

0.8194 

r 

0.9590 

) 

0.500 

0.9324 

( 

0.8509 

r 

0.9743 

) 

0.600 

0.9517 

( 

0.8774 

f 

0.9847 

) 

0.700 

0.9672 

( 

0.9012 

r 

0.9917 

) 

0.800 

0.9799 

( 

0.9240 

r 

0.9962 

) 

0.900 

0.9905 

( 

0.9480 

r 

0.9989 

) 

0.950 

0.9951 

( 

0.9627 

r 

0.9996 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0177, 

0.5395) 

(0.0056, 

0.4250) 

o 

o 

0.6509) 

(0.0577, 

0.6735) 

(0.0297, 

0.5962) 

(0.1032, 

0.7440) 

(0.3436, 

0.8900) 

(0.2720, 

0.8624) 

(0.4213, 

0.9134) 

(0.7430, 

0.9730) 

(0.6685, 

0.9627) 

(0.8077, 

0.9808) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  4,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0201  0.0336  0.1745  0.7718  1.0000 
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1  2 

34.  89. 

3.  27. 


3 

21. 

12. 


4 

2. 

7. 


5 

3, 

33. 


TPF:  0.0000  0,4024  0.4878  0.6341  0.9634  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A-  1.1938  B=  0.7160 

Z(K)=  -0.7446  0.9365  1.8313  2.0514 

LOGL=  -270.8926 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.1283  B=  0.7053 

Z(K)=  -0.7614  0.9868  1.6973  1.9788 

LOGL=  -269.4543 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX : 


0.0311  0.0109 
0.0109  0.0120 
0.0060  0.0028 
0.0065  -0.0015 
0.0040  -0.0078 
0.0022  -0.0114 


0.0060  0.0065 
0.0028  -0.0015 
0.0129  0.0037 
0.0037  0.0140 
0.0019  0.0125 
0.0011  0.0125 


0.0040  0.0022 

-0.0078  -0.0114 
0.0019  0.0011 

0.0125  0.0125 

0.0273  0.0283 

0.0283  0.0381 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1,0000  0.5610 
0.5610  1.0000 
0.3014  0.2234 
0.3090  -0.1165 
0.1366  -0.4327 
0.0635  -0.5335 


0,3014  0.3090 
0.2234  -0.1165 
1.0000  0.2754 
0.2754  1.0000 
0.1009  0.6388 
0.0480  0.5420 


0.1366  0.0635 

-0.4327  -0.5335 
0.1009  0.0480 

0.6388  0.5420 

1.0000  0.8771 

0.8771  1.0000 


AREA  =  0.8217  STD.  DEV.  (AREA)  =  0.0323 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.2455 

( 

0.1253 

r 

0.4095 

) 

0.010 

0.3040 

( 

0.1756 

/ 

0.4627 

) 

0.020 

0.3743 

( 

0.2422 

/ 

0.5231 

) 

0.030 

0.4213 

( 

0.2898 

r 

0.5623 

) 

0.040 

0.4575 

( 

0.3277 

r 

0.5920 

) 

0.050 

0.4872 

( 

0.3595 

r 

0.6162 

) 

0.060 

0.5125 

( 

0.3869 

f 

0.6369 

) 

0.070 

0.5347 

( 

0.4112 

r 

0.6550 

) 

0.080 

0.5545 

( 

0.4330 

r 

0.6711 

) 

0.090 

0.5724 

( 

0.4527 

r 

0.6857 

) 

0.100 

0.5887 

( 

0.4708 

r 

0.6991 

) 

0.110 

0.6037 

( 

0.4875 

r 

0.7114 

) 

0.120 

0.6177 

( 

0.5030 

r 

0.7229 

) 

0.130 

0.6307 

( 

0.5174 

r 

0.7336 

) 

0.140 

0.6429 

( 

0.5310 

t 

0.7437 

) 

0.150 

0.6544 

( 

0.5437 

r 

0.7532 

) 

0.200 

0.7036 

( 

0.5981 

r 

0.7942 

) 

0.250 

0.7430 

( 

0.6415 

r 

0.8271 

) 

0.300 

0.7760 

( 

0.6777 

r 

0.8545 

) 
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0.400 

0.8289 

( 

0.7362 

t 

0.8976 

) 

0.500 

0.8704 

( 

0.7831 

r 

0.9297 

) 

0.600 

0.9043 

( 

0.8233 

r 

0.9540 

) 

0.700 

0.9329 

( 

0.8596 

r 

0.9724 

) 

0.800 

0.9574 

( 

0.8944 

f 

0.9859 

) 

0.900 

0.9789 

( 

0.9309 

r 

0.9951 

) 

0.950 

0.9890 

( 

0.9526 

r 

0.9982 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0239,  0.3946) 
(0.0448,  0.4725) 
(0.1619,  0.6672) 
(0.7768,  0.9521) 


(0.0091,  0.2956)  (0.0552, 

(0.0216,  0.3831)  (0.0848, 

(0.1114,  0.6059)  (0.2252, 

(0.7049,  0.9342)  (0.8375, 

R  0  C  F  I  T  (JUNE  1993  VERSION) 


0.5009) 

0.5633) 

0.7244) 

0.9658) 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  4,  FAS/AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT’ 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 


RESPONSE  DATA: 


CATEGORY 

1 

2 

3 

4 

5 

ACTUALLY  NEGATIVE  CASES 

22. 

58. 

19. 

26. 

24 

ACTUALLY  POSITIVE  CASES 

3. 

6. 

3. 

0. 

15 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.1611  0.3356 

0.4631 

0.8523 

1.0000 

TPF:  0.0000  0.5556  0.5556 

0.6667 

0.8889 

1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  0.5715  B=  0.5571 

Z(K)=  -1.0466  0.0924  0.4241  0.9900 

LOGL=  -260.0469 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.5870  B=  0.5037 

Z(K)=  -1.0604  0.1050  0.4573  0.9660 

LOGL=  -259.7316 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX : 

A  0.0526  0.0065  0.0041  0.0035  0.0036  0.0040 
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1 


B 

0.0065 

0.0229 

0.0031 

0.0000 

-0.0010 

-0.0032 

Z( 

1) 

0.0041 

0.0031 

0.0159 

0.0051 

0.0039 

0.0027 

Z( 

2) 

0.0035 

0.0000 

0.0051 

0.0104 

0.0082 

0.0063 

Z( 

3) 

0.0036 

-0.0010 

0.0039 

0.0082 

0.0111 

0.0086 

Z( 

4) 

0.0040 

-0.0032 

0.0027 

0.0063 

0.0086 

0.0148 

A 

CORRELATION  MATRIX: 
1.0000  0.1880  0.1413 

0.1485 

0.1479 

0.1443 

B 

0.1880 

1.0000 

0.1599 

0.0013 

-0.0638 

-0.1762 

Z( 

1) 

0.1413 

0.1599 

1.0000 

0.3950 

0.2940 

0.1776 

Z( 

2) 

0.1485 

0.0013 

0.3950 

1.0000 

0.7640 

0.5067 

Z( 

3) 

0.1479 

-0.0638 

0.2940 

0.7640 

1.0000 

0.6694 

Z( 

4) 

0.1443 

-0.1762 

0.1776 

0.5067 

0.6694 

1.0000 

AREA  = 

0.6999 

STD. 

.  DEV. (AREA)  = 

0.0700 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.2386 

( 

0.0642 

r 

0.5396 

) 

0.010 

0.2793 

( 

0.0911 

f 

0.5652 

) 

0.020 

0.3272 

( 

0.1285 

r 

0.5940 

) 

0.030 

0.3592 

( 

0.1568 

r 

0.6128 

) 

0.040 

0.3840 

( 

0.1802 

r 

0.6272 

) 

0.050 

0.4045 

( 

0.2006 

r 

0.6391 

) 

0.060 

0.4222 

( 

0.2187 

r 

0.6495 

) 

0.070 

0.4378 

( 

0.2352 

r 

0.6586 

) 

0.080 

0.4519 

( 

0.2504 

r 

0.6670 

) 

0.090 

0.4647 

( 

0.2645 

r 

0.6746 

) 

0.100 

0.4766 

( 

0.2776 

r 

0.6817 

) 

0.110 

0.4877 

( 

0.2901 

r 

0,6884 

) 

0.120 

0.4980 

( 

0.3018 

f 

0.6947 

) 

0.130 

0.5078 

( 

0.3130 

f 

0.7008 

) 

0.140 

0.5171 

( 

0.3236 

r 

0.7065 

) 

0.150 

0.5259 

( 

0.3338 

r 

0.7120 

) 

0.200 

0.5648 

( 

0.3790 

r 

0.7371 

) 

0.250 

0.5977 

( 

0.4173 

r 

0.7592 

) 

0.300 

0.6267 

( 

0.4506 

r 

0.7794 

) 

0.400 

0.6771 

( 

0.5072 

r 

0.8162 

) 

0.500 

0.7214 

( 

0.5547 

r 

0.8500 

) 

0.600 

0.7625 

( 

0.5968 

f 

0.8817 

) 

0.700 

0.8026 

( 

0.6363 

r 

0.9120 

) 

0.800 

0.8439 

( 

0.6763 

r 

0.9411 

) 

0.900 

0.8911 

( 

0.7230 

r 

0.9695 

) 

0.950 

0.9216 

( 

0.7558 

r 

0.9838 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.1670,  0.5400) 


(0.1143,  0.4922)  (0.2333,  0.5872) 
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(0.3237,  0.6393) 
(0.4582,  0.7034) 
(0.8555,  0.8689) 


(0.2533,  0.5996)  (0.4011, 

(0.3804,  0.6677)  (0.5377, 

(0.7919,  0.8405)  (0.9045, 

R  0  C  F  I  T  (JUNE  1993  VERSION) 


0.6775) 

0.7371) 

0.8936) 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  4,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT’ 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  27.  90. 

CASES  7 .  45 . 


3  4  5 

32.  0.  0. 

61.  25.  12. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.2148  0.8188  1.0000 
TPF:  0.0000  0.0800  0.2467  0.6533  0.9533  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  0.9927  B=  0.7668 

Z(K)=  -0.9107  0.7898  2.6112  2.7112 

LOGL=  -384.2195 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.9878  B=  0.6626 

Z(K)=  -0.9335  0.8314  2.5567  3.6335 

LOGL=  -347.6042 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.0162 

0.0042 

0.0057 

0.0058 

0.0026 

-0.0026 

B 

0.0042 

0.0074 

0.0031 

-0.0018 

-0.0188 

-0.0317 

Z( 

1) 

0.0057 

0.0031 

0.0143 

0.0038 

-0.0029 

-0.0079 

Z( 

2) 

0.0058 

-0.0018 

0.0038 

0.0126 

0.0148 

0.0178 

Z( 

3) 

0.0026 

-0.0188 

-0.0029 

0.0148 

0.0832 

0.1112 

Z( 

4) 

-0.0026 

-0.0317 

“0.0079 

0.0178 

0.1112 

0.2013 

A 

CORRELATION  MATRIX: 
1.0000  0.3869  0.3724 

0.4085 

0.0700  -0.0449 

B 

0.3869 

1.0000 

0,2980  • 

“0.1812 

-0.7584  -0.8210 

Z( 

1) 

0.3724 

0.2980 

1.0000 

0.2866 

-0.0835  -0.1477 

Z( 

2) 

0.4085 

-0.1812 

0.2866 

1.0000 

0.4581  0.3540 

Z( 

3) 

0.0700 

-0.7584 

-0.0835 

0.4581 

1.0000  0.8593 

Z( 

4) 

-0.0449 

-0.8210 

-0.1477 

0.3540 

0.8593  1.0000 

AREA  = 

0.7949 

STD 

.  DEV. (AREA)  =  0.0279 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
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FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.2360 

( 

0.1295 

r 

0.3783 

) 

0.010 

0.2898 

( 

0.1764 

r 

0.4291 

) 

0.020 

0.3545 

( 

0.2378 

f 

0.4868 

) 

0.030 

0.3979 

( 

0.2817 

r 

0.5241 

) 

0.040 

0.4315 

( 

0.3167 

r 

0.5525 

) 

0.050 

0.4593 

( 

0.3463 

r 

0.5756 

) 

0.060 

0.4830 

( 

0.3720 

r 

0.5953 

) 

0.070 

0.5039 

( 

0.3949 

r 

0.6126 

) 

0.080 

0.5226 

( 

0.4156 

r 

0.6280 

) 

0.090 

0.5395 

( 

0.4344 

f 

0.6419 

) 

0.100 

0.5551 

( 

0.4518 

r 

0.6547 

) 

0.110 

0.5695 

( 

0.4680 

r 

0.6666 

) 

0.120 

0.5829 

( 

0.4830 

r 

0.6776 

) 

0.130 

0.5954 

( 

0.4971 

r 

0.6879 

) 

0.140 

0.6072 

( 

0.5105 

f 

0.6977 

) 

0.150 

0.6183 

( 

0.5230 

f 

0.7069 

) 

0.200 

0.6665 

( 

0.5775 

r 

0.7470 

) 

0.250 

0.7058 

( 

0.6218 

/ 

0.7800 

) 

0.300 

0.7391 

( 

0.6592 

f 

0.8082 

) 

0.400 

0.7940 

( 

0.7203 

r 

0.8547 

) 

0.500 

0.8384 

( 

0.7698 

/ 

0.8921 

) 

0.600 

0.8760 

( 

0.8123 

/ 

0.9229 

) 

0.700 

0.9091 

( 

0.8507 

r 

0.9485 

) 

0.800 

0.9389 

( 

0.8876 

r 

0.9698 

) 

0.900 

0.9669 

( 

0.9263 

r 

0.9870 

) 

0.950 

0.9811 

( 

0.9493 

r 

0.9941 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0001, 

(0.0053, 

(0.2029, 

(0.8247, 

0.0778) 

0.2400) 

0.6689) 

0.9459) 

(0.0000, 

(0.0009, 

(0.1466, 

(0.7579, 

0.0226) 

0.1399) 

0.6146) 

0.9267) 

(0.0029, 

(0.0232, 

(0.2704, 

(0.8785, 

0.2013) 

0.3701) 

0.7199) 

0.9609) 
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1 


R  0  C  F  I  T  (JUNE  1993  VERSION) 


MAXIMUM  LIKELIH 
OF  A  BINORMA 
FROM  RATI 


OOD  ESTIMATIO 
L  ROC  CURVE 
N  G  DATA 


N 


DATA  DESCRIPTION:  Reader  5,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALir 
NO,  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  59. 


RESPONSE  DATA; 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  104.  33. 

CASES  5.  10. 


3  4  5 

7.  3.  2. 

2.  13.  29. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0134  0.0336  0.0805  0.3020  1.0000 
TPF;  0.0000  0.4915  0.7119  0.7458  0.9153  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.7869  B=  0.7670 

2;(K)-  0.5182  1.4017  1.8313  2.2142 

LOGL=  -210.7154 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.7799  B=  0.7652 

Z(K)=  0.5196  1.4144  1.6996  2.3413 

LOGL=  -207.9325 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


VARIANCE-COVARIANCE  MATRIX : 


A 

0.0959 

0.0452 

0.0101 

0.0059 

0.0006 

-0.0215 

B 

0.0452 

0.0331 

0.0029 

-0.0047 

-0.0105 

-0.0319 

Z( 

1) 

0.0101 

0.0029 

0.0116 

0.0074 

0.0065 

0.0043 

Z( 

2) 

0.0059 

-0,0047 

0.0074 

0.0205 

0.0198 

0.0212 

Z( 

3) 

0.0006 

-0.0105 

0.0065 

0.0198 

0.0282 

0.0323 

Z( 

4) 

-0.0215 

-0.0319 

0.0043 

0.0212 

0.0323 

0.0723 

CORRELATION  MATRIX: 

A 

1.0000  0.8029 

0.3034 

0.1335 

0.0116 

-0.2584 

B 

0.8029  1.0000 

0.1497  ■ 

-0.1790 

-0.3442 

-0.6523 

Z( 

1) 

0.3034  0.1497 

1.0000 

0.4809 

0.3579 

0.1499 

Z( 

2) 

0.1335  -0.1790 

0.4809 

1.0000 

0.8239 

0.5521 

Z( 

3) 

0.0116  -0.3442 

0.3579 

0,8239 

1.0000 

0.7148 

Z( 

4) 

-0.2584  -0.6523 

0.1499 

0.5521 

0.7148 

1.0000 

AREA  -  0.9213 

STD 

.  DEV. (AREA)  = 

0.0241 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.4241 

( 

0.2255 

f 

0.6448 

) 

0.010 

0.4998 

( 

0.3092 

r 

0.6905 

) 

0.020 

0.5824 

( 

0.4096 

r 

0.7405 

) 

0.030 

0.6333 

( 

0.4741 

r 

0.7722 

) 

0.040 

0.6701 

( 

0.5212 

r 

0.7958 

) 

0.050 

0.6988 

( 

0.5580 

r 

0.8149 

) 

0.060 

0.7224 

( 

0.5879 

r 

0.8310 

) 

0.070 

0.7423 

( 

0.6128 

r 

0.8448 

) 

0.080 

0.7595 

( 

0.6340 

r 

0.8570 

) 

0.090 

0.7745 

( 

0.6524 

f 

0.8678 

) 

0.100 

0.7879 

( 

0.6686 

f 

0.8775 

) 

0.110 

0.7999 

( 

0.6829 

r 

0.8862 

) 

0.120 

0.8108 

( 

0.6958 

r 

0.8942 

) 

0.130 

0.8207 

( 

0.7075 

r 

0.9015 

) 

0.140 

0.8298 

( 

0.7181 

r 

0.9081 

) 

0.150 

0.8381 

( 

0.7278 

r 

0.9143 

) 

0.200 

0.8720 

( 

0.7668 

r 

0.9387 

) 

0.250 

0.8969 

( 

0.7955 

r 

0.9557 

) 

0.300 

0.9160 

( 

0.8182 

t 

0.9678 

) 

0.400 

0.9437 

( 

0.8529 

r 

0.9832 

) 

0.500 

0.9625 

( 

0.8796 

r 

0.9915 

) 

0.600 

0.9758 

( 

0.9018 

r 

0.9960 

) 

0.700 

0.9854 

( 

0.9216 

r 

0.9984 

) 

0.800 

0.9923 

( 

0.9403 

f 

0.9995 

) 

0.900 

0,9971 

( 

0.9599 

r 

0.9999 

) 

0.950 

0.9988 

( 

0.9717 

r 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0096, 

0.4954) 

(0.0021, 

0.3391) 

(0.0348, 

0.6524) 

(0.0446, 

0.6842) 

(0.0213, 

0.5901) 

(0.0852, 

0.7676) 

(0.0786, 

0.7573) 

(0.0450, 

0.6855) 

(0.1284, 

0.8192) 

(0.3017, 

0.9166) 

(0.2324, 

0.8889) 

(0.3789, 

0.9387) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  5,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0537  0.0738  0.0940  0.2215  1.0000 
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1 

116. 

3. 


2  3 

19.  3. 

27.  12. 


4 

3. 

7. 


5 

8, 

33, 


TPF:  0.0000  0.4024  0.4878  0.6341  0.9634  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  3.6757  B=  2.4979 

Z(K)=  0.7670  1.3170  1.4482  1.6104 

LOGL=  -227.2513 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  3.6478  B=  2.4614 

Z(K)=  0.7655  1.3378  1.4848  1.5864 

LOGL=  -225.6613 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.4865 

0.3474 

0.0421 

0.0053 

-0.0113 

-0.0232 

B 

0.3474 

0.3153 

0.0132 

-0.0285 

-0.0443 

-0.0555 

Z( 

1) 

0.0421 

0.0132 

0.0131 

0.0099 

0.0091 

0.0086 

Z( 

2) 

0.0053 

-0.0285 

0.0099 

0.0184 

0.0194 

0.0203 

Z( 

3) 

-0.0113 

-0.0443 

0.0091 

0.0194 

0.0226 

0.0239 

Z( 

4) 

-0.0232 

-0.0555 

0.0086 

0.0203 

0.0239 

0.0265 

A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.8870 
0.8870  1.0000 
0.5283  0.2060 
0.0564  -0.3742 
0.1078  -0.5251 
0.2046  -0.6069 


0.5283  0.0564 
0.2060  -0.3742 
1.0000  0.6380 
0.6380  1.0000 
0.5312  0.9540 
0.4623  0.9176 


-0.1078  -0.2046 
-0.5251  -0.6069 
0.5312  0.4623 

0.9540  0.9176 

1.0000  0.9757 

0.9757  1.0000 


AREA  =  0.9151  STD.  DEV. (AREA)  =  0.0197 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.0035 

( 

0.0000 

r 

0.1705 

) 

0.010 

0.0188 

( 

0.0002 

r 

0.2774 

) 

0.020 

0.0795 

( 

0.0042 

r 

0.4267 

) 

0.030 

0.1629 

( 

0.0203 

r 

0.5328 

) 

0.040 

0.2539 

( 

0.0531 

r 

0.6145 

) 

0.050 

0.3439 

( 

0.1018 

r 

0.6801 

) 

0.060 

0.4286 

( 

0.1624 

r 

0.7339 

) 

0.070 

0.5058 

( 

0.2299 

r 

0.7789 

) 

0.080 

0.5748 

( 

0.2996 

r 

0.8167 

) 

0.090 

0.6357 

( 

0.3681 

r 

0.8487 

) 

0.100 

0.6890 

( 

0.4330 

r 

0.8759 

) 

0.110 

0.7351 

( 

0.4930 

r 

0.8988 

) 

0.120 

0.7750 

( 

0.5474 

r 

0.9180 

) 

0.130 

0.8092 

( 

0.5961 

r 

0.9341 

) 

0.140 

0.8386 

( 

0.6394 

r 

0.9474 

) 

0.150 

0.8636 

( 

0.6777 

r 

0.9584 

) 

0.200 

0.9426 

( 

0.8115 

r 

0.9884 

) 

0.250 

0.9766 

( 

0.8851 

r 

0.9972 

) 

0.300 

0.9908 

( 

0.9281 

r 

0.9994 

) 
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0.400 

0.9988 

( 

0.9711 

r 

1.0000 

) 

0.500 

0.9999 

( 

0.9887 

r 

1.0000 

) 

0.600 

1.0000 

( 

0.9960 

r 

1.0000 

) 

0.700 

1.0000 

( 

0.9988 

t 

1.0000 

) 

0.800 

1.0000 

( 

0.9998 

r 

1.0000 

) 

0.900 

1.0000 

( 

1.0000 

r 

1.0000 

) 

0.950 

1.0000 

( 

1.0000 

f 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0563,  0.3986) 
(0.0688,  0.4972) 
(0.0905,  0.6387) 
(0.2220,  0.9611) 


(0.0283,  0.1485)  (0.1026, 

(0.0376,  0.2320)  (0.1170, 

(0.0544,  0.3825)  (0.1418, 

(0.1612,  0.8872)  (0.2941, 

R  0  C  F  I  T  (JUNE  1993  VERSION) 


0.7016) 

0.7636) 

0.8434) 

0.9897) 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  5,  FAS/AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT’ 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 


RESPONSE  DATA: 


CATEGORY 

1 

2 

3 

4 

5 

ACTUALLY  NEGATIVE  CASES 

111. 

26. 

4. 

6. 

2. 

ACTUALLY  POSITIVE  CASES 

13. 

4. 

2. 

3. 

5. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0134  0.0537 

0.0805 

0.2550 

1.0000 

TPF:  0.0000  0.1852  0.2963 

0.3704 

0.5185 

1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  0.4690  B=  0.6101 

Z(K)=  0.6584  1.4017  1.6104  2.2142 

LOGL=  -158.0757 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  3  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.4488  B=  0.5997 

Z(K)=  0.6599  1.3843  1.6234  2.2257 

LOGL=  -158.0101 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ; 


VARIANCE-COVARIANCE  MATRIX: 

A  0.1054  0.0456  0.0090  0.0031  0.0000  -0.0132 
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B 

0.0456 

0.0390 

0.0026 

-0.0041 

-0.0076 

-0.0227 

Z( 

1) 

0.0090 

0.0026 

0.0124 

0.0086 

0.0077 

0.0058 

Z( 

2) 

0.0031 

-0.0041 

0.0086 

0.0206 

0.0193 

0.0181 

Z( 

3) 

0.0000 

-0.0076 

0.0077 

0.0193 

0.0273 

0.0261 

Z( 

4) 

-0.0132 

-0.0227 

0.0058 

0.0181 

0.0261 

0.0689 

A 

CORRELATION  MATRIX: 
1.0000  0.7112  0.2506 

0.0663 

0.0009 

-0.1547 

B 

0.7112 

1.0000 

0.1173  - 

■0.1452 

-0,2333 

-0.4373 

Z( 

1) 

0.2506 

0.1173 

1.0000 

0.5374 

0.4195 

0.1978 

Z( 

2) 

0.0663 

-0.1452 

0.5374 

1.0000 

■  0.8143 

0.4807 

Z( 

3) 

0.0009 

-0.2333 

0.4195 

0.8143 

1.0000 

0.6019 

Z( 

4) 

-0.1547 

-0.4373 

0.1978 

0.4807 

0.6019 

1.0000 

AREA  = 

0.6499 

STD. 

DEV. (AREA)  = 

0.0947 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUFfD) 

0.005 

0.1365 

( 

0.0359 

r 

0.3478 

) 

0.010 

0.1720 

( 

0.0571 

r 

0.3770 

) 

0.020 

0.2168 

( 

0.0891 

r 

0.4131 

) 

0.030 

0.2485 

( 

0.1141 

r 

0.4389 

) 

0.040 

0.2738 

( 

0.1352 

f 

0.4600 

) 

0.050 

0.2954 

( 

0.1535 

t 

0.4785 

) 

0.060 

0.3143 

( 

0.1697 

r 

0.4952 

) 

0.070 

0.3313 

( 

0.1843 

t 

0.5106 

) 

0.080 

0.3468 

( 

0.1975 

f 

0.5250 

) 

0.090 

0.3612 

( 

0.2097 

r 

0.5386 

) 

0.100 

0.3746 

( 

0.2209 

r 

0.5516 

) 

0.110 

0.3871 

( 

0.2313 

r 

0.5640 

) 

0.120 

0.3990 

( 

0.2410 

r 

0.5759 

) 

0.130 

0.4103 

( 

0.2501 

r 

0.5874 

) 

0.140 

0.4211 

( 

0.2586 

r 

0.5986 

) 

0.150 

0.4314 

( 

0.2667 

r 

0.6093 

) 

0.200 

0.4778 

( 

0.3013 

r 

0.6588 

) 

0.250 

0.5178 

( 

0.3292 

t 

0.7023 

) 

0.300 

0.5535 

( 

0.3528 

r 

0.7411 

) 

0.400 

0.6168 

( 

0.3922 

r 

0.8073 

) 

0.500 

0.6732 

( 

0.4256 

r 

0.8611 

) 

0.600 

0.7259 

( 

0.4567 

r 

0.9049 

) 

0.700 

0.7773 

( 

0.4879 

f 

0.9402 

) 

0.800 

0.8298 

( 

0.5222 

r 

0.9679 

) 

0.900 

0.8883 

( 

0.5669 

r 

0.9883 

) 

0.950 

0.9244 

( 

0.6017 

r 

0.9955 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0130,  0.1878) 


(0.0031,  0.1162) 
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(0.0435,  0.2818) 


(0.0523,  0.2999)  (0.0258,  0.2362)  (0.0968,  0.3705) 

(0.0831,  0.3515)  (0.0479,  0.2911)  (0.1351,  0.4159) 

(0.2547,  0.5212)  (0.1900,  0.4691)  (0.3292,  0.5729) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  5,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT: 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0336  0.1275  0.4430  1.0000 
TPF:  0.0000  0.1733  0.4667  0.5800  0.9200  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.4291  B=  0.8815 

Z(K)=  0.1432  1.1383  1.8313  2.7112 

LOGL=  -384.7947 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.4421  B=  0.9120 

Z(K)=  0.1247  1.2686  1.6829  2.6384 

LOGL=  -378.6452 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0313  0.0148  0.0098  0.0061  0.0023  -0.0103 

B  0.0148  0.0167  0.0034  -0.0058  -0.0116  -0.0288 

Z(  1)  0.0098  0.0034  0.0105  0.0058  0.0044  0.0010 

Z(  2)  0.0061  -0.0058  0.0058  0.0165  0.0175  0.0226 

Z(  3)  0.0023  -0.0116  0.0044  0.0175  0.0255  0.0356 

Z(  4)  -0.0103  -0.0288  0.0010  0.0226  0.0356  0.0777 


CORRELATION  MATRIX: 

A  1.0000  0.6487  0.5393  0.2689  0.0797  -0.2097 

B  0.6487  1.0000  0.2560  -0.3462  -0.5634  -0.7976 

Z(  1)  0.5393  0.2560  1.0000  0.4366  0.2684  0.0361 

Z(  2)  0.2689  -0.3462  0.4366  1.0000  0.8534  0.6315 

Z(  3)  0.0797  -0.5634  0.2684  0.8534  1.0000  0.7996 

Z(  4)  -0.2097  -0.7976  0.0361  0.6315  0.7996  1.0000 


AREA  =  0.8567  STD.  DEV. (AREA)  =  0.0228 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
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1 

83. 

12. 


2 

47. 

51. 


3 

14. 

17. 


4 

5. 

44. 


5 

0. 

26. 


FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1821 

( 

0.0792 

f 

0.3429 

) 

0.010 

0.2483 

( 

0.1292 

r 

0.4092 

) 

0.020 

0.3331 

( 

0.2039 

r 

0.4861 

) 

0.030 

0.3922 

( 

0.2616 

r 

0.5364 

) 

0.040 

0.4384 

( 

0.3093 

r 

0.5746 

) 

0.050 

0.4767 

( 

0.3502 

r 

0.6057 

) 

0.060 

0.5095 

( 

0.3860 

r 

0.6321 

) 

0.070 

0.5382 

( 

0.4179 

r 

0.6551 

) 

0.080 

0.5637 

( 

0.4466 

f 

0.6755 

) 

0.090 

0.5867 

( 

0.4726 

r 

0.6939 

) 

0.100 

0.6076 

( 

0.4964 

r 

0.7107 

) 

0.110 

0.6268 

( 

0.5183 

r 

0.7261 

) 

0.120 

0.6445 

( 

0.5385 

r 

0.7403 

) 

0.130 

0.6608 

( 

0.5572 

r 

0.7535 

) 

0.140 

0.6761 

( 

0.5747 

f 

0.7658 

) 

0,150 

0.6904 

( 

0.5911 

r 

0.7774 

) 

0.200 

0.7501 

( 

0.6593 

r 

0.8261 

) 

0.250 

0.7960 

( 

0.7115 

r 

0.8636 

) 

0.300 

0.8325 

( 

0.7532 

r 

0.8932 

) 

0.400 

0.8871 

( 

0.8166 

r 

0.9358 

) 

0.500 

0.9254 

( 

0.8633 

r 

0.9632 

) 

0.600 

0.9528 

( 

0.9000 

f 

0.9805 

) 

0.700 

0.9726 

( 

0.9300 

r 

0.9910 

) 

0.800 

0.9864 

( 

0.9552 

r 

0.9967 

) 

0.900 

0.9955 

( 

0.9773 

r 

0.9994 

) 

0.950 

0.9984 

( 

0.9877 

r 

0.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0042, 

0.1675) 

(0.0007, 

0.0718) 

(0.0182, 

0.3207) 

(0.0462, 

0.4631) 

(0.0230, 

0.3527) 

(0.0853, 

0.5763) 

(0.1023, 

0.6122) 

(0.0642, 

0.5221) 

(0.1546, 

0.6967) 

(0.4504, 

0.9080) 

(0.3723, 

0.8739) 

(0.5305, 

0.9347) 
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R  0  C  F  I  T  (JUNE  1993  VERSION) 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  6,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALITY 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  59. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 
CASES  51.  88. 

CASES  4 .  6 . 


3  4  5 

6.  2.  2. 

8.  13.  28. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0134  0.0268  0.0671  0.6577  1.0000 
TPF:  0.0000  0.4746  0.6949  0.8305  0.9322  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3876  B=  0.5072 

Z(K) - 0.4058  1,4979  1.9297  2.2142 

LOGL=  -231.0280 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  Y 


PROCEDURE  CONVERGES  AFTER  9  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.5292  B=  0.5732 

Z(K)=  -0.3860  1.3887  1.9468  2.6902 

LOGL=  -223.1701 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  Y 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.0608 

0.0217 

0.0050 

0.0064 

0.0019 

-0.0134 

B 

0.0217 

0.0157 

0.0018 

-0.0021 

-0.0091 

-0.0270 

Z( 

1) 

0.0050 

0.0018 

0.0111 

0.0037 

0.0025 

0.0006 

Z( 

2) 

0.0064 

-0.0021 

0.0037 

0.0207 

0.0189 

0.0202 

Z( 

3) 

0.0019 

-0.0091 

0.0025 

0.0189 

0.0394 

0.0462 

Z( 

4) 

-0.0134 

-0.0270 

0.0006 

0.0202 

0.0462 

0.1115 

A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.7016 
0.7016  1.0000 
0.1930  0.1340 
0.1810  -0.1160 
0.0391  -0.3673 
0.1623  -0.6461 


0.1930  0.1810 
0.1340  -0.1160 
1.0000  0.2411 
0.2411  1.0000 
0.1213  0.6625 
0.0172  0.4205 


0.0391  -0.1623 
-0.3673  -0.6461 
0.1213  0.0172 

0.6625  0.4205 

1.0000  0.6968 

0.6968  1.0000 


AI^A  =  0.9077  STD.  DEV. (AREA)  =  0.0284 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.5209 

( 

0.3446 

¥ 

0.6932 

) 

0.010 

0.5775 

( 

0.4129 

¥ 

0.7293 

) 

0.020 

0.6375 

( 

0.4879 

¥ 

0.7684 

) 

0.030 

0.6739 

( 

0.5339 

¥ 

0.7929 

) 

0.040 

0.7004 

( 

0.5672 

¥ 

0.8110 

) 

0.050 

0.7211 

( 

0.5932 

¥ 

0.8255 

) 

0.060 

0.7382 

( 

0.6144 

¥ 

0.8376 

) 

0.070 

0.7527 

( 

0.6323 

¥ 

0.8481 

) 

0.080 

0.7654 

( 

0.6477 

¥ 

0.8572 

) 

0.090 

0.7765 

( 

0.6613 

¥ 

0.8654 

) 

0.100 

0.7865 

( 

0.6734 

¥ 

0.8728 

) 

0.110 

0.7956 

( 

0.6842 

¥ 

0.8795 

) 

0.120 

0.8039 

( 

0.6941 

¥ 

0.8856 

) 

0.130 

0.8115 

( 

0.7032 

¥ 

0.8913 

) 

0.140 

0.8186 

( 

0.7115 

¥ 

0.8965 

) 

0.150 

0.8251 

( 

0.7192 

¥ 

0.9014 

) 

0.200 

0.8524 

( 

0.7511 

¥ 

0.9216 

) 

0.250 

0.8734 

( 

0.7755 

¥ 

0.9368 

) 

0.300 

0.8904 

( 

0.7954 

¥ 

0.9487 

) 

0.400 

0.9168 

( 

0.8270 

¥ 

0.9661 

) 

0.500 

0.9369 

( 

0.8522 

¥ 

0.9779 

) 

0.600 

0.9529 

( 

0.8740 

¥ 

0.9862 

) 

0.700 

0.9663 

( 

0.8941 

¥ 

0.9920 

) 

0.800 

0.9779 

( 

0.9141 

¥ 

0.9961 

) 

0.900 

0.9882 

( 

0.9364 

¥ 

0.9987 

) 

0.950 

0.9933 

( 

0.9509 

¥ 

0.9995 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0036,  0.4949) 
(0.0258,  0.6603) 
(0.0825,  0.7683) 
(0.6503,  0.9600) 


(0.0004, 
(0.0098, 
(0.0474, 
(0.5713, 
R  0  C  F  I 


0.3490) 

0.5755) 

0.7161) 

0.9487) 

'  (JUNE  1993 


(0.0209, 

(0.0596, 

(0.1343, 

(0.7232, 

VERSION) 


0.6414) 

0.7376) 

0.8146) 

0.9692) 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  6,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALITY 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 


RESPONSE  DATA: 


CATEGORY 

1 

2  3 

4 

5 

ACTUALLY  NEGATIVE  CASES 

46. 

87.  8. 

5. 

3. 

ACTUALLY  POSITIVE  CASES 

6. 

11.  5. 

14. 

46. 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0201  0.0537 

0.1074 

0.6913  1.0000 
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TPF:  0.0000  0.5610  0.7317  0.7927  0.9268  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2746  B=  0.4676 

Z(K)-  -0.4991  1.2407  1.6104  2.0514 

LOGL=  -260.5021 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.3109  B=  0.4987 

Z(K)=  -0.4857  1.1876  1.5601  2.2769 

LOGL-  -257.8493 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.0341  0.0109 

0.0109  0.0093 

0.0044  0.0017 

0.0046  -0.0015 
0.0036  -0.0040 
-0.0012  -0.0132 


0.0044  0.0046 
0.0017  -0.0015 
0.0114  0.0038 
0.0038  0.0168 
0.0030  0.0154 
0.0012  0.0153 


0.0036  -0.0012 
-0.0040  -0.0132 
0.0030  0.0012 

0.0154  0.0153 

0.0239  0.0250 

0.0250  0.0613 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.6111 
0.6111  1.0000 
0.2221  0.1647 

0.1916  -0.1178 
0.1275  -0.2656 
0.0269  -0.5533 


0.2221  0.1916 
0.1647  -0.1178 
1.0000  0.2735 
0.2735  1.0000 
0.1818  0.7662 
0,0462  0.4750 


0.1275  -0.0269 
-0.2656  -0.5533 
0.1818  0.0462 

0.7662  0,4750 

1.0000  0.6528 

0,6528  1,0000 


AREA  =  0.8796  STD.  DEV. (AREA)  =  0.0285 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.5104 

( 

0.3577 

f 

0.6616 

) 

0.010 

0.5598 

( 

0.4169 

r 

0.6953 

) 

0.020 

0.6127 

( 

0.4819 

r 

0.7318 

) 

0,030 

0.6453 

( 

0.5224 

r 

0.7547 

) 

0.040 

0.6692 

( 

0.5520 

r 

0.7717 

) 

0.050 

0.6881 

( 

0.5754 

r 

0.7855 

) 

0.060 

0.7038 

( 

0.5947 

f 

0.7970 

) 

0.070 

0.7173 

( 

0.6112 

r 

0.8071 

) 

0.080 

0.7291 

( 

0.6255 

r 

0.8160 

) 

0.090 

0.7396 

( 

0.6382 

r 

0.8240 

) 

0.100 

0.7491 

( 

0.6496 

t 

0.8313 

) 

0.110 

0.7578 

( 

0.6599 

r 

0.8379 

) 

0.120 

0.7657 

( 

0.6694 

r 

0.8441 

) 

0.130 

0.7731 

( 

0.6781 

t 

0.8499 

) 

0.140 

0.7800 

( 

0.6861 

r 

0.8553 

) 

0.150 

0.7864 

( 

0.6936 

r 

0.8603 

) 

0.200 

0.8136 

( 

0.7250 

r 

0.8819 

) 

0.250 

0.8351 

( 

0.7495 

r 

0.8991 

) 

0.300 

0.8530 

( 

0.7697 

r 

0.9133 

) 
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0.400 

0.8819 

( 

0.8022 

r 

0.9357 

) 

0.500 

0.9051 

( 

0.8286 

f 

0.9528 

) 

0.600 

0.9246 

( 

0.8517 

r 

0.9664 

) 

0.700 

0.9421 

( 

0.8733 

f 

0.9774 

) 

0.800 

0.9582 

( 

0.8951 

r 

0.9863 

) 

0.900 

0.9744 

( 

0.9200 

f 

0.9937 

) 

0.950 

0.9835 

( 

0.9366 

r 

0.9969 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER  BOUND 

UPPER 

BOUND 

(  fpf  , 

TPF  ) 

(  FPF  ,  TPF  ) 

(  FPF  , 

TPF  ) 

(0.0114, 

0.5696) 

(0.0029,  0.4735) 

(0.0366, 

0.6618) 

(0.0594, 

0.7029) 

(0.0312,  0.6487) 

(0.1043, 

0.7530) 

(0.1175, 

0.7638) 

(0.0747,  0.7230) 

(0.1753, 

0.8011) 

(0.6864, 

0.9398) 

(0.6088,  0.9263) 

(0.7565, 

0.9513) 

R 

0  C  F  I  T  (JUNE  1993 

i  VERSION) 

: 

MAXI 

MUM 

L 

I 

K  E 

L 

IHOOD  EST 

I 

M  A  T  I  0 

N 

0 

F  A 

B 

I 

N  0 

R 

M  A  L  ROC  C 

U 

R  V  E 

F  R 

0 

M 

R 

A 

TING  DATA 

DATA  DESCRIPTION:  Reader  6,  FAS/AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149. 


NO.  OF  ACTUALLY  POSITIVE  CASES 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  42.  87. 

CASES  2 .  9 . 


3  4  5 

7.  12.  1. 

5.  7.  4. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0067  0.0872  0.1342  0.7181  1.0000 
TPF:  0.0000  0.1481  0.4074  0.5926  0.9259  1.0000 


27. 


INITIAL  VALUES  OF  PARAMETERS: 

A=  0.9964  B=  0.8218 

Z(K)=  -0.5769  1.1067  1.3581  2.4728 

LOGL=  -198.7468 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ; 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.0463  B=  0.8553 

Z(K)=  -0.5722  1.0813  1.4054  2.4435 

LOGL=  -198.2965 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0833  0.0310  0.0072  0.0082  0.0068  -0.0055 
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B 

0.0310 

0.0312 

0.0029 

-0.0019 

-0.0051 

-0.0270 

Z( 

1) 

0.0072 

0.0029 

0.0118 

0.0037 

0.0031 

0.0008 

Z( 

2) 

0.0082 

-0.0019 

0.0037 

0.0156 

0.0140 

0.0129 

Z( 

3) 

0.0068 

-0.0051 

0.0031 

0.0140 

0.0207 

0.0199 

Z( 

4) 

-0.0055 

-0.0270 

0.0008 

0.0129 

0.0199 

0.0889 

CORRELATION  MATRIX: 

A 

1.0000  0.6084 

0.2307 

0.2287 

0.1637 

-0.0639 

B 

0.6084  1.0000 

0.1536  ■ 

-0.0877 

-0.2004 

-0.5135 

Z( 

1) 

0,2307  0.1536 

1.0000 

0.2763 

0.1981 

0.0251 

Z( 

2) 

0.2287  -0.0877 

0.2763 

1.0000 

0.7777 

0.3453 

Z( 

3) 

0.1637  -0.2004 

0.1981 

0.7777 

1.0000 

0.4639 

Z( 

4) 

-0.0639  -0.5135 

0.0251 

0.3453 

0.4639 

1.0000 

AREA  =  0.7867 

STD 

.  DEV. (AREA)  = 

0.0540 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


0.005 

0.1236 

( 

0.0311 

r 

0.3266 

) 

0.010 

0.1726 

( 

0.0562 

f 

0.3819 

) 

0.020 

0.2386 

( 

0.0985 

r 

0.4479 

) 

0.030 

0.2868 

( 

0.1343 

r 

0.4924 

) 

0.040 

0.3258 

( 

0.1657 

f 

0.5272 

) 

0.050 

0.3591 

( 

0.1940 

r 

0.5562 

) 

0.060 

0.3883 

( 

0.2198 

r 

0.5813 

) 

0.070 

0.4144 

( 

0.2435 

r 

0.6036 

) 

0.080 

0.4381 

( 

0.2655 

/ 

0.6237 

) 

0.090 

0.4599 

( 

0.2859 

r 

0.6421 

) 

0.100 

0.4801 

( 

0.3051 

r 

0.6590 

) 

0.110 

0.4988 

( 

0.3231 

r 

0.6748 

) 

0.120 

0.5164 

( 

0.3401 

r 

0.6895 

) 

0.130 

0.5330 

( 

0.3563 

r 

0.7034 

) 

0.140 

0.5487 

( 

0.3716 

r 

0.7164 

) 

0.150 

0.5635 

( 

0.3861 

r 

0.7288 

) 

0.200 

0.6280 

( 

0.4501 

r 

0.7819 

) 

0.250 

0.6807 

( 

0.5029 

r 

0.8244 

) 

0.300 

0.7251 

( 

0.5479 

r 

0.8590 

) 

0.400 

0.7967 

( 

0.6228 

r 

0.9110 

) 

0.500 

0.8523 

( 

0.6846 

f 

0.9465 

) 

0.600 

0.8966 

( 

0.7388 

f 

0.9703 

) 

0.700 

0.9325 

( 

0.7891 

r 

0.9856 

) 

0.800 

0.9613 

( 

0.8384 

r 

0.9945 

) 

0.900 

0.9839 

( 

0.8917 

r 

0.9989 

) 

0.950 

0.9929 

( 

0.9244 

r 

0.9997 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0073,  0.1483) 


(0.0012,  0.0613)  (0.0315,  0.2933) 
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(0.0800,  0.4381)  (0.0458,  0.3457)  (0.1306,  0.5340) 

(0.1398,  0.5483)  (0.0924,  0.4650)  (0.2014,  0.6296) 

(0.7164,  0.9377)  (0.6402,  0.9120)  (0.7839,  0.9571) 

R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  6,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0067  0.1208  0.9329  1.0000 
TPF:  0.0000  0.1000  0.3133  0.7867  0.9933  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3830  B=  0.8297 

Z(K)=  -1.4979  1.1711  2.4728  2.7112 

LOGL=  -313.1151 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.6109  B=  0.7475 

Z(K)=  -1.4729  1.1370  2.7862  3.8593 

LOGL=  -285.8086 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0461  0.0232  0.0070  0.0102  -0.0279  -0.0598 
B  0.0232  0.0210  0.0041  -0.0013  -0.0444  -0.0753 
Z(  1)  0.0070  0.0041  0.0238  0.0026  -0.0055  -0.0113 
Z(  2)  0.0102  -0.0013  0.0026  0.0165  0.0181  0.0200 
Z(  3)  -0.0279  -0.0444  -0.0055  0.0181  0.1331  0.1946 
Z(  4)  -0.0598  -0.0753  -0.0113  0.0200  0.1946  0.3304 


CORRELATION  MATRIX: 

A  1.0000  0.7445  0.2125  0.3717  -0.3564  -0.4846 
B  0.7445  1.0000  0.1815  -0.0677  -0.8396  -0.9048 
Z(  1)  0.2125  0.1815  1.0000  0.1322  -0.0969  -0.1278 
Z(  2)  0.3717  -0.0677  0.1322  1.0000  0.3857  0.2706 
Z(  3)  -0.3564  -0.8396  -0.0969  0.3857  1.0000  0.9278 
Z(  4)  -0.4846  -0.9048  -0.1278  0.2706  0.9278  1.0000 


AREA  =  0.9015  STD.  DEV. (AREA)  =  0.0210 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
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1  2  3  4  5 

10.  121.  17.  1.  0. 

1.  31.  71.  32.  15. 


FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.3764 

( 

0.2065 

/ 

0.5748 

) 

0.010 

0.4489 

( 

0.2827 

r 

0.6248 

) 

0.020 

0.5300 

( 

0.3767 

f 

0,6790 

) 

0.030 

0.5811 

( 

0.4394 

r 

0.7129 

) 

0.040 

0.6187 

( 

0.4865 

r 

0.7382 

) 

0.050 

0.6484 

( 

0.5243 

r 

0.7585 

) 

0.060 

0.6731 

( 

0.5555 

t 

0.7756 

) 

0.070 

0.6941 

( 

0.5821 

r 

0.7904 

) 

0.080 

0.7124 

( 

0.6051 

r 

0.8035 

) 

0.090 

0.7286 

( 

0.6254 

r 

0.8153 

) 

0.100 

0.7431 

( 

0.6433 

r 

0.8260 

) 

0.110 

0.7561 

( 

0.6594 

r 

0.8357 

) 

0.120 

0.7681 

( 

0.6739 

r 

0.8448 

) 

0.130 

0.7790 

( 

0.6872 

r 

0.8531 

) 

0.140 

0.7891 

( 

0.6993 

r 

0.8609 

) 

0.150 

0.7985 

( 

0.7104 

r 

0.8682 

) 

0.200 

0.8369 

( 

0.7553 

r 

0.8984 

) 

0.250 

0.8658 

( 

0.7883 

r 

0.9212 

) 

0.300 

0.8886 

( 

0.8142 

r 

0.9388 

) 

0.400 

0.9225 

( 

0.8534 

r 

0.9635 

) 

0.500 

0.9464 

( 

0.8830 

r 

0.9789 

) 

0.600 

0.9641 

( 

0.9071 

r 

0.9886 

) 

0.700 

0.9774 

( 

0.9281 

t 

0.9945 

) 

0.800 

0.9875 

( 

0.9474 

r 

0.9979 

) 

0.900 

0.9949 

( 

0.9668 

r 

0.9995 

) 

0.950 

0.9977 

( 

0.9779 

r 

0.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0001, 

0.1013) 

(0.0000, 

0.0172) 

(0.0031, 

0.3330) 

(0.0027, 

0.3185) 

(0-0002, 

0.1571) 

(0.0192, 

0.5250) 

(0.1278, 

0.7767) 

(0.0825, 

0.7166) 

(0.1880, 

0.8287) 

(0.9296, 

0.9967) 

(0.8791, 

0.9935) 

(0.9621, 

0.9983) 
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R  0  C  F  I  T  (JUNE  1993  VERSION) 


MAXIMUM  LIKELIH 
OF  A  BINORMA 
FROM  RATI 


OOD  ESTIMATION 
L  ROC  CURVE 
N  G  DATA 


DATA  DESCRIPTION:  Reader  7,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALITY 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  59. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 
CASES  82.  51. 

CASES  1 .  8 . 


3  4  5 

8.  6.  2. 

4.  9.  37. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0134  0.0537  0.1074  0.4497  1.0000 
TPF:  0.0000  0.6271  0.7797  0.8475  0.9831  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  2.1865  B=  0.8676 

Z(K)=  0.1262  1.2407  1.6104  2.2142 

LOGL=  -220.2996 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  Y 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  2.1233  B=  0.8333 

Z(K)=  0.1238  1.2597  1.6061  2.1666 

LOGL=  -220.1214 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.1270  0.0611 

0.0611  0.0430 

0.0083  0.0026 

0.0099  -0.0018 
0.0041  -0.0086 
-0.0168  -0.0286 


0.0083  0.0099 
0.0026  -0.0018 
0.0106  0.0055 
0.0055  0.0180 
0.0046  0.0165 
0.0032  0.0161 


0.0041  -0.0168 
-0.0086  -0.0286 
0.0046  0.0032 

0.0165  0.0161 

0.0253  0.0269 

0.0269  0.0560 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.8274 
0.8274  1.0000 
0.2270  0.1239 
0.2065  -0.0647 
0.0715  -0.2618 
0.1987  -0.5828 


0.2270  0.2065 
0.1239  -0.0647 
1.0000  0.4000 
0.4000  1.0000 
0.2837  0.7723 
0.1313  0.5057 


0.0715  -0.1987 
-0.2618  -0.5828 
0.2837  0.1313 

0.7723  0.5057 

1.0000  0.7138 

0.7138  1.0000 


AREA  =  0.9486  STD.  DEV. (AREA)  =  0.0174 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.4907 

( 

0.2628 

/ 

0.7217 

) 

0.010 

0.5732 

( 

0.3620 

t 

0.7648 

) 

0.020 

0.6597 

( 

0.4772 

r 

0.8107 

) 

0.030 

0.7108 

( 

0.5483 

r 

0.8389 

) 

0.040 

'  0.7467 

( 

0.5986 

r 

0.8596 

) 

0.050 

0.7741 

( 

0.6367 

r 

0.8759 

) 

0.060 

0.7960 

( 

0.6669 

f 

0.8894 

) 

0.070 

0.8142 

( 

0.6916 

r 

0.9008 

) 

0.080 

0.8295 

( 

0.7122 

r 

0.9106 

) 

0.090 

0.8428 

( 

0.7298 

r 

0.9192 

) 

0.100 

0.8543 

( 

0.7450 

r 

0.9267 

) 

0.110 

0.8646 

( 

0.7583 

r 

0.9334 

) 

0.120 

0.8737 

( 

0.7701 

r 

0.9393 

) 

0.130 

0.8819 

( 

0.7806 

f 

0.9446 

) 

0.140 

0.8893 

( 

0.7901 

r 

0.9494 

) 

0.150 

0.8961 

( 

0.7987 

r 

0.9537 

) 

0.200 

0.9225 

( 

0.8325 

r 

0.9699 

) 

0.250 

0.9408 

( 

0.8566 

f 

0.9802 

) 

0.300 

0.9542 

( 

0.8751 

r 

0.9869 

) 

0.400 

0.9721 

( 

0.9026 

r 

0.9943 

) 

0.500 

0.9831 

( 

0.9229 

f 

0.9976 

) 

0.600 

0.9902 

( 

0.9392 

f 

0.9991 

) 

0.700 

0.9948 

( 

0.9532 

r 

0.9997 

) 

0.800 

0.9976 

( 

0.9660 

r 

0.9999 

) 

0.900 

0.9993 

( 

0.9785 

r 

1.0000 

) 

0.950 

0.9998 

( 

0.9856 

t 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS; 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0151, 

0.6247) 

(0.0043, 

0.4726) 

(0.0443, 

0.7594) 

(0.0541, 

0.7837) 

(0.0275, 

0.7002) 

(0.0978, 

0.8520) 

(0.1039, 

0.8585) 

(0.0639, 

0.8035) 

(0.1595, 

0.9020) 

(0.4507, 

0.9783) 

(0.3725, 

0.9680) 

(0.5310, 

0.9857) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  7,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  82. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0336  0.0671  0.0805  0.3423  1.0000 

98 


1  2 

98.  39. 

5.  7. 


3 

2. 

0. 


4 

5. 

8. 


5 

5. 

62. 


TPF:  0.0000  0.7561  0.8537  0.8537  0.9390  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A-  1.8145  B=  0.5476 

Z(K)=  0.4058  1.4017  1.4979  1.8313 

LOGL=  -205.7691 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.8307  B=  0.5978 

Z(K)-  0.4095  1.3735  1.4355  1.8921 

LOGL=  -204.5984 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.0774  0.0364 
0.0364  0.0270 
0.0075  0.0024 
0.0040  -0.0042 
0.0032  -0.0051 
0.0060  -0.0154 


0.0075  0.0040 
0.0024  -0.0042 
0.0112  0.0067 
0.0067  0.0202 
0.0065  0.0199 
0.0051  0.0195 


0.0032 

-0.0051 

0.0065 

0.0199 

0.0215 

0.0212 


-0.0060 

-0.0154 

0.0051 

0.0195 

0.0212 

0.0394 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.7951 
0.7951  1.0000 
0.2546  0.1361 
0.1018  -0.1777 
0.0795  -0.2116 
0,1083  -0.4724 


0.2546  0.1018 
0.1361  -0.1777 
1.0000  0.4468 
0.4468  1.0000 
0.4199  0.9569 
0.2418  0.6916 


0.0795 

-0.2116 

0.4199 

0.9569 

1.0000 

0.7298 


-0.1083 

-0.4724 

0.2418 

0.6916 

0.7298 

1.0000 


AREA  =  0.9420  STD.  DEV. (AREA)  =  0.0190 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.6144 

(  0.4107 

r 

0.7902 

) 

0.010 

0.6700 

(  0.4929 

r 

0.8153 

) 

0.020 

0.7267 

(  0.5796 

f 

0.8425 

) 

0.030 

0.7600 

(  0.6305 

r 

0.8598 

) 

0.040 

0.7835 

(  0.6658 

r 

0.8728 

) 

0.050 

0.8016 

(  0.6924 

f 

0.8833 

) 

0.060 

0.8162 

(  0.7135 

r 

0.8923 

) 

0.070 

0.8285 

(  0.7307 

r 

0.9000 

) 

0.080 

0.8391 

(  0.7452 

r 

0.9069 

) 

0.090 

0.8483 

(  0.7576 

r 

0.9131 

) 

0.100 

0.8565 

(  0.7683 

t 

0.9186 

) 

0.110 

0.8638 

(  0.7778 

t 

0.9237 

) 

0.120 

0.8704 

(  0.7862 

r 

0.9283 

) 

0.130 

0.8764 

(  0.7937 

f 

0.9326 

) 

0.140 

0.8820 

(  0.8005 

r 

0.9365 

) 

0.150 

0.8871 

(  0.8067 

t 

0.9402 

) 

0.200 

0.9079 

(  0.8313 

t 

0.9551 

) 

0.250 

0.9233 

(  0.8491 

r 

0.9658 

) 

0.300 

0.9354 

(  0.8630 

f 

0.9739 

) 
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0.400 

0.9535 

(  0.8843 

r 

0.9847 

) 

0.500 

0.9664 

(  0.9007 

r 

0.9912 

) 

0.600 

0.9763 

(  0.9147 

¥ 

0.9953 

) 

0.700 

0.9840 

(  0.9274 

r 

0.9977 

) 

0.800 

0.9902 

(  0.9401 

r 

0.9991 

) 

0.900 

0.9953 

(  0.9543 

¥ 

0.9998 

) 

0.950 

0.9976 

(  0.9637 

¥ 

0.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0292,  0.7580) 
(0.0756,  0.8346) 
(0.0848,  0.8437) 
(0.3411,  0.9436) 


(0.0113,  0.6798) 
(0.0425,  0.7884) 
(0.0493,  0.8005) 
(0.2687,  0.9281) 
R  0  C  F  I  T  (JUNE 


(0.0664,  0.8244) 
(0.1255,  0.8738) 
(0.1368,  0.8802) 
(0.4199,  0.9564) 
VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  7,  FAS /AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 
CASES  87.  40. 

CASES  5 .  7 . 


3  4  5 

6.  12.  4. 

1.  8.  6. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0268  0.1074  0.1477  0.4161  1.0000 
TPF:  0.0000  0.2222  0.5185  0.5556  0.8148  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.1369  B=  0.9553 

Z(K)=  0.2115  1.0466  1.2407  1.9297 

LOGL=  -203.8137 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.1181  B=  0.9444 

Z(K)=  0.2135  1.0427  1.2129  1.9635 

LOGL=  -203.6756 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX : 

A  0.1028  0.0501  0.0105  0.0077  0.0063  -0.0052 


100 


B 

0.0501 

0.0503 

0.0036 

-0.0028 

-0.0051 

-0.0218 

Z( 

1) 

0.0105 

0.0036 

0.0107 

0.0066 

0.0061 

0.0040 

Z( 

2) 

0.0077 

-0.0028 

0.0066 

0.0149 

0.0140 

0.0123 

Z( 

3) 

0.0063 

-0.0051 

0.0061 

0.0140 

0.0171 

0.0154 

Z( 

4) 

-0.0052 

-0.0218 

0.0040 

0.0123 

0.0154 

0.0438 

CORRELATION  MATRIX: 

A 

1.0000  0.6973 

0.3172 

0.1970 

0.1497 

-0.0771 

B 

0.6973  1.0000 

0.1548 

-0.1016 

-0.1732 

-0.4642 

Z( 

1) 

0.3172  0.1548 

1.0000 

0.5240 

0.4478 

0.1836 

Z( 

2) 

0.1970  -0.1016 

0.5240 

1.0000 

0.8780 

0.4818 

Z( 

3) 

0.1497  -0.1732 

0.4478 

0.8780 

1.0000 

0.5624 

Z( 

4) 

-0.0771  -0.4642 

0.1836 

0.4818 

0.5624 

1.0000 

AREA  =  0.7919 

STD 

.  DEV. (AREA)  = 

0.0521 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.0943 

( 

0.0161 

r 

0.3129 

) 

0.010 

0.1402 

( 

0.0346 

r 

0.3663 

) 

0.020 

0.2056 

( 

0.0709 

¥ 

0.4306 

) 

0,030 

0.2551 

( 

0.1050 

f 

0.4746 

) 

0.040 

0.2961 

( 

0.1368 

¥ 

0.5094 

) 

0.050 

0.3316 

( 

0.1664 

¥ 

0.5388 

) 

0.060 

0.3630 

( 

0.1940 

¥ 

0.5645 

) 

0.070 

0.3913 

( 

0.2197 

¥ 

0.5876 

) 

0.080 

0.4172 

( 

0.2439 

¥ 

0.6087 

) 

0.090 

0.4411 

( 

0.2665 

¥ 

0.6281 

) 

0.100 

0.4632 

( 

0.2878 

¥ 

0.6462 

) 

0.110 

0.4839 

( 

0.3078 

¥ 

0.6632 

) 

0.120 

0.5033 

( 

0.3268 

¥ 

0.6793 

) 

0.130 

0.5217 

( 

0.3447 

¥ 

0.6944 

) 

0.140 

0.5390 

( 

0.3616 

¥ 

0.7088 

) 

0.150 

0.5554 

( 

0.3777 

¥ 

0.7225 

) 

0.200 

0.6268 

( 

0.4474 

¥ 

0.7821 

) 

0.250 

0.6849 

( 

0.5036 

¥ 

0.8299 

) 

0.300 

0.7334 

( 

0.5506 

¥ 

0.8685 

) 

0.400 

0.8104 

( 

0.6265 

¥ 

0.9245 

) 

0.500 

0.8682 

( 

0.6878 

¥ 

0.9596 

) 

0.600 

0.9126 

( 

0.7410 

¥ 

0.9807 

) 

0.700 

0.9466 

( 

0.7901 

¥ 

0.9922 

) 

0.800 

0.9721 

( 

0.8384 

¥ 

0.9977 

) 

0.900 

0.9901 

( 

0.8907 

¥ 

0.9997 

) 

0.950 

0.9962 

( 

0.9231 

¥ 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0248,  0.2308) 


(0.0088,  0.1306)  (0.0602,  0.3636) 
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(0.1126,  0.4891)  (0.0709,  0.3939)  (0.1694,  0.5850) 

(0.1485,  0.5531)  (0.1000,  0.4633)  (0.2108,  0.6402) 

(0.4155,  0.8203)  (0.3386,  0.7658)  (0.4957,  0.8660) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  7,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALITY 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  149. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  150. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 
CASES  43.  87. 

CASES  2.  24. 


3  4  5 

19.  0.  0. 

89.  24.  11. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.1275  0.7114  1.0000 
TPF:  0.0000  0.0733  0.2333  0.8267  0.9867  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.8003  B=  1.0541 

Z(K)=  -0.5571  1.1383  2.6112  2.7112 

LOGL=  -343.1591 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  Y 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.7862  B=  0.7355 

Z(K)=  -0.5591  1.1430  3.4209  4.4039 

LOGL=  -311.2273 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  Y 


VARIANCE-COVARIANCE  MATRIX : 


A 

0.0407 

0.0213 

0.0064 

0.0068 

-0.0468 

-0.0744 

B 

0.0213 

0.0236 

0.0032 

-0.0046 

-0.0791 

-0.1112 

Z( 

1) 

0.0064 

0.0032 

0.0118 

0.0036 

-0.0060 

-0.0102 

Z( 

2) 

0.0068 

-0.0046 

0.0036 

0.0168 

0.0303 

0.0364 

Z( 

3) 

-0.0468 

-0.0791 

-0.0060 

0.0303 

0.3152 

0.4174 

Z( 

4) 

-0.0744 

-0.1112 

-0.0102 

0.0364 

0.4174 

0.5974 

CORRELATION  MATRIX: 


A 

1.0000 

0.6882 

0.2914 

0.2582 

-0.4137 

-0.4775 

B 

0.6882 

1.0000 

0.1894 

-0.2296 

-0.9163 

-0.9358 

Z( 

1) 

0.2914 

0.1894 

1.0000 

0.2550 

-0.0979 

-0.1216 

Z( 

2) 

0.2582 

-0.2296 

0.2550 

1.0000 

0.4159 

0.3636 

Z( 

3) 

-0.4137 

-0.9163 

-0.0979 

0.4159 

1.0000 

0.9618 

Z( 

4) 

-0.4775 

-0.9358 

-0.1216 

0.3636 

0.9618 

1.0000 

V  AREA  =  0.9249  STD.  DEV. (AREA)  =  0.0167 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
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FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.4568 

( 

0.2456 

r 

0.6814 

) 

0.010 

0,5299 

( 

0.3296 

r 

0.7227 

) 

0.020 

0.6085 

( 

0,4307 

/ 

0.7659 

) 

0.030 

0.6564 

( 

0.4966 

/ 

0.7922 

) 

0.040 

0.6909 

( 

0.5455 

r 

0.8112 

) 

0.050 

0.7178 

( 

0.5842 

f 

0.8264 

) 

0.060 

0.7397 

( 

0.6160 

r 

0.8390 

) 

0.070 

0,7582 

( 

0.6428 

f 

0.8498 

) 

0.080 

0.7742 

( 

0.6658 

r 

0.8593 

) 

0.090 

0.7882 

( 

0.6859 

r 

0.8677 

) 

0.100 

0.8006 

( 

0.7037 

f 

0.8754 

) 

0.110 

0.8117 

( 

0.7194 

r 

0.8824 

) 

0.120 

0.8217 

( 

0.7336 

f 

0.8888 

) 

0.130 

0.8309 

( 

0.7464 

r 

0.8948 

) 

0.140 

0.8393 

( 

0.7580 

r 

0.9003 

) 

0.150 

0.8471 

( 

0.7686 

r 

0.9055 

) 

0.200 

0.8785 

( 

0.8106 

r 

0.9271 

) 

0.250 

0.9015 

( 

0.8402 

/ 

0.9436 

) 

0.300 

0.9194 

( 

0.8626 

r 

0.9563 

) 

0.400 

0.9452 

( 

0.8949 

r 

0.9743 

) 

0.500 

0.9630 

( 

0.9179 

r 

0.9854 

) 

0.600 

0.9757 

( 

0.9359 

f 

0.9923 

) 

0.700 

0.9851 

( 

0.9510 

r 

0.9964 

) 

0.800 

0.9919 

( 

0.9646 

r 

0.9987 

) 

0.900 

0.9968 

( 

0.9778 

t 

0,9997 

) 

0.950 

0.9986 

( 

0.9853 

r 

0.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0000, 

(0.0003, 

(0.1265, 

(0.7120, 

0.0732) 

0.2328) 

0.8278) 

0.9860) 

(0.0000, 

(0.0000, 

(0.0812, 

(0.6355, 

0.0051) 

0.0619) 

0.7760) 

0.9794) 

(0.0019, 

(0.0102, 

(0.1870, 

(0.7799, 

0.3675) 

0.5317) 

0.8713) 

0.9907) 
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R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  9,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL: 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  125. 


NO.  OF  ACTUALLY  POSITIVE  CASES 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  87 .  25 . 

CASES  4 .  6 . 


3  4  5 

13.  0.  0. 

10.  9.  22. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.1040  0.3040  1.0000 
TPF:  0.0000  0.4314  0.6078  0.8039  0.9216  1.0000 


51. 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.7240  B=  0.6484 

Z(K)=  0.5125  1.2592  2.5525  2.6525 

LOGL=  -186.1320 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR] 


PROCEDURE  CONVERGES  AFTER  8  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.5837  B=  0.4840 

Z(K)=  0.5062  1.3070  2.7322  3.6470 

LOGL=  -175.6608 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR] 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.0885 

0.0335 

0.0074 

0.0054 

-0.0349 

-0.0885 

B 

0.0335 

0.0230 

0.0020 

-0.0021 

-0.0488 

-0.0949 

Z( 

1) 

0.0074 

0.0020 

0.0138 

0.0091 

0.0041 

0.0004 

Z( 

2) 

0.0054 

-0.0021 

0.0091 

0.0226 

0.0227 

0.0268 

Z( 

3) 

-0.0349 

-0.0488 

0.0041 

0.0227 

0.2101 

0.2978 

Z( 

4) 

-0.0885 

-0.0949 

0.0004 

0.0268 

0.2978 

0.5556 

A 

CORRELATION  MATRIX: 
1.0000  0.7424  0.2112 

0.1205 

-0.2558 

-0.3991 

B 

0.7424 

1.0000 

0.1114 

-0.0941 

-0.7011 

-0.8390 

Z( 

1) 

0.2112 

0.1114 

1.0000 

0.5152 

0.0759 

0.0042 

Z( 

2) 

0.1205 

-0.0941 

0.5152 

1.0000 

0.3294 

0.2388 

Z( 

3) 

-0.2558 

-0.7011 

0.0759 

0.3294 

1.0000 

0.8716 

Z( 

4) 

-0.3991 

-0.8390 

0.0042 

0.2388 

0.8716 

1.0000 

AREA  = 

0.9230 

STD 

.  DEV.  (AREA)  = 

0.0307 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.6318 

( 

0.4299 

r 

0.8024 

) 

0.010 

0.6763 

( 

0.4955 

r 

0.8228 

) 

0.020 

0.7222 

( 

0.5636 

/ 

0.8458 

) 

0.030 

0.7496 

( 

0.6035 

r 

0.8608 

) 

0.040 

0.7692 

( 

0.6315 

9 

0.8721 

) 

0.050 

0.7845 

( 

0.6527 

9 

0.8814 

) 

0.060 

0.7970 

( 

0.6698 

9 

0.8893 

) 

0.070 

0.8076 

( 

0.6839 

9 

0.8961 

) 

0.080 

0.8169 

( 

0.6959 

9 

0.9022 

) 

0.090 

0.8250 

( 

0.7063 

9 

0.9077 

) 

0.100 

0.8323 

( 

0.7154 

9 

0.9127 

) 

0.110 

0.8389 

( 

0.7236 

9 

0.9172 

) 

0.120 

0.8449 

( 

0.7309 

9 

0.9214 

) 

0.130 

0.8505 

( 

0.7375 

9 

0.9252 

) 

0.140 

0.8556 

( 

0.7436 

9 

0.9288 

) 

0.150 

0.8604 

( 

0.7492 

9 

0.9322 

) 

0.200 

0.8803 

( 

0.7719 

9 

0.9460 

) 

0.250 

0.8957 

( 

0.7890 

9 

0.9565 

) 

0.300 

0.9082 

( 

0.8027 

9 

0.9648 

) 

0.400 

0.9280 

( 

0.8242 

9 

0.9768 

) 

0.500 

0.9434 

( 

0.8415 

9 

0.9849 

) 

0.600 

0.9560 

( 

0.8566 

9 

0.9905 

) 

0.700 

0.9669 

( 

0.8709 

9 

0.9945 

) 

0.800 

0.9768 

( 

0.8857 

9 

0.9973 

) 

0.900 

0.9862 

( 

0.9034 

9 

0.9991 

) 

0.950 

0.9913 

( 

0.9160 

9 

0.9996 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0001, 

0.4280) 

(0.0000, 

0.1871) 

(0.0144, 

0.7004) 

(0.0031, 

0.6030) 

(0.0001, 

0.4311) 

(0.0333, 

0.7568) 

(0.0956, 

0.8292) 

(0.0546, 

0.7906) 

(0.1557, 

0.8630) 

(0.3064, 

0.9097) 

(0.2308, 

0.8902) 

(0.3912, 

0.9265) 

1  R  0  C  F  I  T  (JUNE  1993  VERSION)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  9,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL! 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  125.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  65. 


RESPONSE  DATA: 


CATEGORY 

1 

2  3 

4 

5 

ACTUALLY  NEGATIVE 

CASES 

120. 

4.  0. 

0. 

1 

ACTUALLY  POSITIVE 

CASES 

23. 

0.  2. 

5. 

35 

OBSERVED  OPERATING 

POINTS : 

FPF:  0.0000  0.0080 

1  0.0080 

0.0080  0. 

.0400  1.0000 
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TPF:  0.0000  0.5385  0.6154  0.6462  0.6462  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3348  B=  0.4725 

Z(K)=  1.7511  2.2093  2.3093  2.4093 

LOGL=  -98.6914 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR] 


PROCEDURE  CONVERGES  AFTER  8  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.8911  B=  0.2887 

Z(K)-  1.7590  2.0535  2.2508  2.7339 

LOGL=  -94.1625 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR] 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


VARIANCE-COVARIANCE  MATRIX: 


0.1439  0.0582 
0.0582  0.0291 
0.0187  0.0038 
0.0051  -0.0093 
0.0258  -0.0206 
0.0947  -0.0582 


0.0187  -0.0051 
0.0038  -0.0093 
0.0419  0.0377 
0.0377  0.0578 
0.0351  0.0596 
0.0291  0.0693 


-0.0258  -0.0947 
-0.0206  -0.0582 
0.0351  0.0291 

0.0596  0.0693 

0.0815  0.1048 

0.1048  0.2247 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.9004 
0.9004  1.0000 
0.2413  0.1084 
0.0564  -0.2260 
0.2380  -0.4226 
0.5266  -0.7202 


0.2413  -0.0564 
0.1084  -0.2260 
1.0000  0.7662 
0.7662  1.0000 
0.6009  0.8682 
0.2999  0.6084 


-0.2380  -0.5266 
-0.4226  -0.7202 
0.6009  0.2999 

0.8682  0.6084 

1.0000  0.7746 

0.7746  1.0000 


AREA  =  0.8040 


STD.  DEV. (AREA)  -  0.0912 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUNE 

L 

UPPER  BOUND) 

0.005 

0.5585 

( 

0.4096 

r 

0.6995 

) 

0.010 

0.5868 

( 

0.4516 

r 

0.7123 

) 

0.020 

0.6171 

( 

0.4897 

r 

0.7330 

) 

0.030 

0.6360 

( 

0.5087 

r 

0.7498 

) 

0.040 

0.6501 

( 

0.5204 

f 

0.7642 

) 

0.050 

0.6613 

( 

0.5284 

r 

0.7767 

) 

0.060 

0.6708 

( 

0.5341 

r 

0.7877 

) 

0.070 

0.6790 

( 

0.5384 

r 

0.7976 

) 

0.080 

0.6863 

( 

0.5418 

r 

0.8066 

) 

0.090 

0.6928 

( 

0.5446 

r 

0.8148 

) 

0.100 

0.6988 

( 

0.5468 

r 

0.8224 

) 

0.110 

0.7043 

( 

0.5486 

r 

0.8294 

) 

0.120 

0.7094 

( 

0.5501 

r 

0,8358 

) 

0.130 

0.7142 

( 

0.5515 

f 

0.8419 

) 

0.140 

0.7188 

( 

0.5526 

r 

0.8476 

) 

0.150 

0.7230 

( 

0.5535 

9 

0.8529 

) 

0.200 

0.7415 

( 

0.5567 

9 

0.8756 

) 

0.250 

0.7569 

( 

0.5584 

9 

0.8936 

) 

0.300 

0.7703 

( 

0.5592 

9 

0.9083 

) 
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0.400 

0.7933 

( 

0.5595 

r 

0.9314 

) 

0.500 

0.8136 

( 

0.5587 

r 

0.9489 

) 

0.600 

0.8325 

( 

0.5571 

r 

0.9628 

) 

0.700 

0.8514 

( 

0.5549 

r 

0.9742 

) 

0.800 

0.8716 

( 

0.5517 

f 

0.9837 

) 

0.900 

0.8964 

( 

0.5465 

r 

0.9919 

) 

0.950 

0.9140 

( 

0.5417 

r 

0.9957 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER  BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  ,  TPF  ) 

(  FPF  , 

TPF  ) 

(0.0031, 

0.5405) 

(0.0001,  0.4339) 

(0.0355, 

0.6443) 

(0.0122, 

0.5953) 

(0.0025,  0.5317) 

(0.0454, 

0.6564) 

(0.0200, 

0.6172) 

(0.0058,  0.5644) 

(0.0568, 

0.6679) 

(0.0393, 

0.6492) 

(0.0154,  0.6054) 

(0.0872, 

0.6911; 

R 

0 

C  F  I  T  (JUNE  1993 

VERSION)  : 

MAXI 

MUM 

L 

I 

K  E 

L 

IHOOD  EST 

I  M  A  T  I  0 

N 

0 

F  A 

B 

I 

N  0 

R 

M  A  L  ROC  C 

U  R  V  E 

F  R 

0 

M 

R 

A 

TING  DATA 

DATA  DESCRIPTION:  Reader  9,  FAS/AD 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL! 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  125.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  27. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  110.  10. 

CASES  14.  2. 


3  4  5 

4.  0.  1. 

2.  6.  3. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0080  0.0080  0.0400  0.1200  1.0000 
TPF:  0.0000  0.1111  0.3333  0.4074  0.4815  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  0.9060  B=  0.7267 

Z(K)=  1.1751  1.7511  2.3093  2.4093 

LOGL=  -104.1775 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR] 


PROCEDURE  CONVERGES  AFTER  9  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.7393  B=  0.6325 

Z(K)=  1.1833  1.6786  2.1371  2.9004 

LOGL=  -96.8623 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR] 


VARIANCE-COVARIANCE  MATRIX: 

A  0.2046  0.0948  0.0185  0.0021  -0.0236  -0.1023 
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1 


B 

0.0948 

0.0614 

0.0043 

-0.0083 

-0.0279 

-0.0864 

Z( 

1) 

0.018'5 

0.0043 

0.0213 

0.0172 

0.0144 

0.0097 

Z( 

2) 

0.0021 

-0.0083 

0.0172 

0.0338 

0.0328 

0.0381 

Z( 

3) 

-0.0236 

-0.0279 

0.0144 

0.0328 

0.0676 

0.0869 

Z( 

4) 

-0.1023 

-0.0864 

0.0097 

0.0381 

0.0869 

0.2634 

A 

CORRELATION  MATRIX: 
1.0000  0.8458  0.2800 

0.0251 

-0.2005 

-0.4407 

B 

0.8458  1.0000 

0.1192  • 

-0.1824 

-0.4333 

-0.6794 

Z( 

1) 

0.2800  0.1192 

1.0000 

0.6440 

0.3785 

0.1302 

Z( 

2) 

0.0251  -0.1824 

0.6440 

1.0000 

0.6866 

0.4042 

Z( 

3) 

-0.2005  -0.4333 

0.3785 

0.6866 

1.0000 

0.6512 

Z( 

4) 

-0.4407  -0.6794 

0.1302 

0.4042 

0.6512 

1.0000 

AREA  =  0.7339 

STD 

.  DEV. (AREA)  = 

0.1068 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1867 

( 

0.0572 

/ 

0.4202 

) 

0.010 

0.2319 

( 

0.0903 

r 

0.4499 

) 

0.020 

0.2877 

( 

0.1370 

r 

0.4895 

) 

0.030 

0.3261 

( 

0.1708 

t 

0.5199 

) 

0.040 

0.3563 

( 

0.1972 

f 

0.5459 

) 

0.050 

0.3816 

( 

0.2186 

t 

0.5691 

) 

0.060 

0.4035 

( 

0.2366 

/ 

0.5904 

) 

0.070 

0.4229 

( 

0.2519 

r 

0.6102 

) 

0.080 

0.4405 

( 

0.2652 

f 

0.6286 

) 

0.090 

0.4566 

( 

0.2769 

9 

0.6459 

) 

0.100 

0.4715 

( 

0.2873 

r 

0.6622 

) 

0.110 

0.4854 

( 

0.2966 

r 

0.6776 

) 

0.120 

0.4984 

( 

0.3051 

r 

0.6921 

) 

0.130 

0.5107 

( 

0.3128 

t 

0.7059 

) 

0.140 

0.5223 

( 

0.3199 

t 

0.7190 

) 

0.150 

0.5334 

( 

0.3264 

r 

0.7315 

) 

0.200 

0.5820 

( 

0.3533 

t 

0.7854 

) 

0.250 

0.6228 

( 

0.3739 

f 

0.8282 

) 

0.300 

0.6583 

( 

0.3908 

r 

0.8628 

) 

0.400 

0.7188 

( 

0.4183 

r 

0.9138 

) 

0.500 

0.7701 

( 

0.4414 

r 

0.9480 

) 

0.600 

0.8157 

( 

0.4628 

r 

0.9707 

) 

0.700 

0.8579 

( 

0.4843 

r 

0.9854 

) 

0.800 

0.8982 

( 

0.5081 

r 

0.9942 

) 

0.900 

0.9394 

( 

0.5393 

r 

0.9987 

) 

0.950 

0.9625 

( 

0.5638 

r 

0.9997 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0019,  0.1367) 


(0.0000,  0.0417)  (0.0291,  0.3231) 
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(0.0163, 

0.2701) 

(0.0041,  0.1749) 

(0.0518, 

0.3859) 

(0.0466, 

0.3735) 

(0.0207,  0.2911) 

(0.0937, 

0.4623) 

(0.1183, 

0.4963) 

(0.0709,  0.4247) 

(0.1847, 

0.5681) 

R 

0 

C  F  I  T  (JUNE  1993 

VERSION)  : 

MAXI 

MUM  L 

I 

K  E 

L 

IHOOD  EST 

I  M  A  T  I  0 

N 

0 

FAB 

I 

N  0 

R 

M  A  L  ROC  C 

U  R  V  E 

FRO 

M 

R 

A 

TING  DATA 

DATA  DESCRIPTION:  Reader  9^  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMAL! 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  125. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  125. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  74.  45. 

CASES  14 .  35 . 


3  4  5 

6.  0.  0. 

54.  10.  12. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.0480  0.4080  1.0000 
TPF:  0.0000  0.0960  0.1760  0.6080  0.8880  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A-  1.5928  B-  1.0021 

Z(K)=  0.2323  1.6649  2.5525  2.6525 

LOGL-  -291.5787 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  AR] 


PROCEDURE  CONVERGES  AFTER  6  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A-  1.3650  B=  0.6517 

Z(K)=  0.2320  1.6703  3.5245  4.0979 

LOGL=  -276.9216 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARJ 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.0333 

0.0153 

0.0090 

0.0013 

-0.0356 

-0.0483 

B 

0.0153 

0.0162 

0.0030 

-0.0129 

-0.0612 

-0.0759 

Z( 

1) 

0.0090 

0.0030 

0.0128 

0.0060 

-0.0025 

-0.0052 

Z( 

2) 

0.0013 

-0.0129 

0.0060 

0.0358 

0.0715 

0.0829 

Z( 

3) 

-0.0356 

-0.0612 

-0.0025 

0.0715 

0.2948 

0.3453 

Z( 

4) 

-0.0483 

-0.0759 

-0.0052 

0.0829 

0.3453 

0.4386 

CORRELATION  MATRIX: 


A 

1.0000 

0.6584 

0.4356 

0.0388 

-0.3594 

-0.4000 

B 

0.6584 

1.0000 

0.2100  - 

-0.5363 

-0.8854 

-0.9004 

Z( 

1) 

0.4356 

0.2100 

1.0000 

0.2790 

-0.0408 

-0.0689 

Z( 

2) 

0.0388  -0.5363 

0.2790 

1.0000 

0.6956 

0.6613 

Z( 

3) 

-0.3594  -0.8854 

-0.0408 

0.6956 

1.0000 

0.9602 

Z( 

4) 

-0.4000  -0.9004 

-0.0689 

0.6613 

0.9602 

1.0000 

AREA  = 

0.8736 

STD, 

.  DEV.  (AREA)  = 

0.0249 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 


no 


FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.3767 

( 

0.2113 

f 

0.5690 

) 

0.010 

0.4398 

( 

0.2780 

r 

0.6125 

) 

0.020 

0.5104 

( 

0.3595 

r 

0.6599 

) 

0.030 

0.5552 

( 

0.4139 

r 

0.6898 

) 

0.040 

0.5885 

( 

0.4553 

r 

0.7122 

) 

0.050 

0.6151 

( 

0.4887 

r 

0.7303 

) 

0.060 

0.6374 

( 

0.5168 

r 

0.7456 

) 

0.070 

0.6565 

( 

0.5410 

r 

0.7590 

) 

0.080 

0.6733 

( 

0.5621 

f 

0.7709 

) 

0.090 

0.6883 

( 

0.5809 

r 

0.7817 

) 

0.100 

0.7018 

( 

0.5978 

f 

0.7915 

) 

0.110 

0.7141 

( 

0.6130 

r 

0.8006 

) 

0.120 

0.7254 

( 

0.6270 

f 

0.8090 

) 

0.130 

0.7359 

( 

0.6397 

r 

0.8170 

) 

0.140 

0.7456 

( 

0.6515 

f 

0.8244 

) 

0.150 

0.7547 

( 

0.6625 

f 

0.8314 

) 

0.200 

0.7929 

( 

0.7075 

f 

0.8615 

) 

0.250 

0.8227 

( 

0.7415 

f 

0.8855 

) 

0.300 

0.8470 

( 

0.7687 

r 

0.9053 

) 

0.400 

0.8850 

( 

0.8107 

t 

0.9357 

) 

0.500 

0.9139 

( 

0.8431 

r 

0.9575 

) 

0.600 

0.9370 

( 

0.8703 

f 

0.9733 

) 

0.700 

0.9560 

( 

0.8947 

r 

0.9847 

) 

0.800 

0.9721 

( 

0.9182 

/ 

0.9925 

) 

0.900 

0.9861 

( 

0.9433 

r 

0.9976 

) 

0.950 

0.9926 

( 

0.9589 

r 

0.9991 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

o 

o 

o 

o 

o 

0.0958) 

(0.0000, 

0.0157) 

(0.0026, 

0.3228) 

(0.0002, 

0.1756) 

(0.0000, 

0.0520) 

(0.0069, 

0.4058) 

(0.0474, 

0.6089) 

(0.0206, 

0.5138) 

(0.0969, 

0.6978) 

(0.4083, 

0.8876) 

(0.3250, 

0.8575) 

(0.4959, 

0.9128) 
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1 

MAXIMUM 
OF  A 
F 


R  0  C  F  I  T 

LIKELIHOOD 
BINORMAL  R 
OM  RATING 


(JUNE  1993  VERSION)  : 

ESTIMATION 
0  C  CURVE 
DATA 


DATA  DESCRIPTION:  Reader  10,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALITY 


NO.  OF  ACTUALLY  NEGATIVE  CASES  -  100. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  42. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  94 .  4 . 

CASES  14.  3. 


3  4  5 

1.  0.  1. 

3.  4.  18. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0100  0.0100  0.0200  0.0600  1.0000 
TPF:  0.0000  0.4286  0.5238  0.5952  0.6667  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A-  1.5622  B=  0.6980 

Z(K)=  1.5551  2.0542  2.2268  2.3268 

LOGL-  -87.2521 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.4407  B=  0.6401 

Z(K)=  1.5592  1.9417  2.2197  2.5170 

LOGL=  -84.9281 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.3827 

0.1857 

0.0387 

-0.0155 

-0.0682 

-0.1359 

B 

0.1857 

0.1039 

0.0084 

-0.0248 

-0.0567 

-0.0972 

Z( 

1) 

0.0387 

0.0084 

0.0400 

0.0346 

0.0311 

0.0273 

Z( 

2) 

-0.0155 

-0.0248 

0.0346 

0.0585 

0.0639 

0.0722 

Z( 

3) 

-0.0682 

-0.0567 

0.0311 

0.0639 

0.0962 

0.1157 

Z( 

4) 

-0.1359 

-0.0972 

0.0273 

0.0722 

0.1157 

0.1722 

A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0 

0.9310  1 

0.3127  0 

-0.1038  -0 
-0.3552  -0 
-0.5292  -0 


9310 

0.3127 

0000 

0.1307 

1307 

1.0000 

3180 

0.7158 

5666 

0.5011 

7266 

0.3296 

-0.1038 

-0.3180 

0.7158 

1.0000 

0.8521 

0.7189 


-0.3552 

-0.5666 

0.5011 

0.8521 

1.0000 

0.8989 


-0.5292 

-0.7266 

0.3296 

0.7189 

0.8989 

1.0000 


AREA-  0.8875  STD.  DEV. (AREA)  =  0.0691 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.4174 

( 

0.1907 

r 

0.6767 

) 

0.010 

0.4806 

( 

0.2717 

r 

0.6951 

) 

0.020 

0.5500 

( 

0.3638 

r 

0.7256 

) 

0.030 

0.5935 

( 

0.4167 

f 

0.7528 

) 

0.040 

0.6254 

( 

0.4508 

f 

0.7773 

) 

0.050 

0.6508 

( 

0.4744 

r 

0.7993 

) 

0.060 

0.6719 

( 

0.4916 

r 

0.8190 

) 

0.070 

0.6900 

( 

0.5046 

r 

0.8365 

) 

0.080 

0.7058 

( 

0.5148 

r 

0.8520 

) 

0.090 

0.7198 

( 

0.5230 

r 

0.8658 

) 

0.100 

0.7324 

( 

0.5297 

r 

0.8782 

) 

0.110 

0.7439 

( 

0.5353 

r 

0.8892 

) 

0.120 

0.7544 

( 

0.5402 

f 

0.8990 

) 

0.130 

0.7641 

( 

0.5443 

f 

0.9079 

) 

0.140 

0.7731 

( 

0.5480 

r 

0.9159 

) 

0.150 

0.7815 

( 

0.5512 

r 

0.9230 

) 

0.200 

0.8165 

( 

0.5630 

t 

0.9501 

) 

0.250 

0.8435 

( 

0.5708 

f 

0.9671 

) 

0.300 

0.8655 

( 

0.5765 

r 

0.9782 

) 

0.400 

0.8995 

( 

0.5845 

r 

0.9905 

) 

0.500 

0.9252 

( 

0.5902 

r 

0.9960 

) 

0.600 

0.9455 

( 

0.5948 

r 

0.9985 

) 

0.700 

0.9621 

( 

0.5989 

r 

0.9995 

) 

0.800 

0.9761 

( 

0.6029 

r 

0.9999 

) 

0.900 

0.9881 

( 

0.6074 

r 

1.0000 

) 

0.950 

0.9937 

( 

0.6105 

r 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0059,  0.4323) 
(0.0132,  0.5079) 
(0.0261,  0.5784) 
(0.0595,  0.6710) 


(0.0004,  0.2447)  (0.0442, 

(0.0023,  0.3559)  (0.0535, 

(0.0078,  0.4579)  (0.0711, 

(0.0255,  0.5760)  (0.1215, 

R  0  C  F  I  T  (JUNE  1993  VERSION) 


0.6369) 

0.6587) 

0.6919) 

0.7560) 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  10,  MicroCalcifications 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 


NO.  OF  ACTUALLY  NEGATIVE  CASES  = 

100. 

NO. 

OF  ACTUALLY  POSITIVE  CASES  = 

RESPONSE  DATA: 

CATEGORY 

1 

2 

3 

4 

5 

ACTUALLY  NEGATIVE  CASES 

91. 

4. 

0. 

1. 

4. 

ACTUALLY  POSITIVE  CASES 

18. 

4. 

0. 

5. 

24. 

OBSERVED  OPERATING  POINTS: 
FPF:  0.0000  0.0400  0.0500 

0.0900 

1.0000 
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TPF:  0.0000  0.4706  0.5686  0.6471  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.7439  B=  1.0039 

Z(K)=  1.3410  1.6452  1.7511 

LOGL=  -99.3411 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ; 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.7844  B=  1.0437 

Z(K)=  1.3425  1.5801  1.7761 

LOGL=  -98.0825 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.6934 

0.4292 

0.0497 

-0.0069 

-0.0612 

B 

0.4292 

0.2914 

0.0129 

-0.0273 

-0.0655 

Z( 

1) 

0.0497 

0.0129 

0.0311 

0.0281 

0.0258 

Z( 

2) 

-0.0069 

-0.0273 

0.0281 

0.0370 

0.0395 

Z( 

3) 

-0.0612 

-0.0655 

0.0258 

0.0395 

0.0529 

A 

CORRELATION  MATRIX: 
1.0000  0.9547  0.3385 

-0.0433 

-0.3193 

B 

0.9547 

1.0000 

0.1350 

-0.2628 

-0.5272 

Z( 

1) 

0.3385 

0.1350 

1.0000 

0.8281 

0.6350 

Z( 

2) 

-0.0433 

-0.2628 

0.8281 

1.0000 

0.8929 

Z( 

3) 

-0.3193 

-0.5272 

0.6350 

0.8929 

1.0000 

AREA  =  0.8915  STD.  DEV. (AREA)  =  0.0515 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1829 

( 

0.0150 

0.6407 

) 

0.010 

0.2598 

( 

0.0475 

/ 

0.6488 

) 

0.020 

0.3596 

( 

0.1264 

r 

0.6644 

) 

0.030 

0.4290 

( 

0.2035 

f 

0.6813 

) 

0.040 

0.4828 

( 

0.2705 

f 

0.7002 

) 

0.050 

0.5268 

( 

0.3258 

r 

0.7211 

) 

0.060 

0.5641 

( 

0.3699 

r 

0.7437 

) 

0.070 

0.5963 

( 

0.4045 

/ 

0.7672 

) 

0.080 

0,6246 

( 

0.4313 

/ 

0.7906 

) 

0.090 

0.6498 

( 

0.4521 

r 

0.8133 

) 

0.100 

0.6725 

( 

0.4684 

r 

0.8346 

) 

0.110 

0.6929 

( 

0.4813 

r 

0.8543 

) 

0.120 

0.7116 

( 

0.4917 

r 

0.8722 

) 

0.130 

0.7287 

( 

0.5002 

f 

0.8882 

) 

0.140 

0.7444 

( 

0.5072 

r 

0.9025 

) 

0.150 

0.7589 

( 

0.5131 

r 

0.9151 

) 

0.200 

0.8176 

( 

0.5321 

r 

0.9584 

) 

0.250 

0.8601 

( 

0.5423 

r 

0.9801 

) 

0.300 

0.8921 

( 

0.5487 

f 

0,9907 

) 

0.400 

0.9358 

( 

0.5563 

r 

0.9981 

) 

0.500 

0.9628 

( 

0.5605 

r 

0.9997 

) 
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0.600 

0.9797 

( 

0.5632  , 

1.0000 

) 

0.700 

0.9901 

( 

0.5650  , 

1.0000 

) 

0.800 

0.9961 

( 

0.5661  , 

1.0000 

) 

0.900 

0.9991 

( 

0.5665  , 

1.0000 

) 

0.950 

0.9998 

( 

0.5663  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER  BOUND 

UPPER 

BOUND 

(  fpf  , 

TPF  ) 

(  FPF  ,  TPF  ) 

(  FPF  , 

TPF  ) 

(0.0379, 

0.4724) 

(0.0130,  0.2946) 

(0.0926, 

0.6559) 

(0.0570, 

0.5538) 

(0.0252,  0.3981) 

(0.1145, 

0.7016) 

(0.0897, 

0.6492) 

(0.0457,  0.5090) 

(0.1594, 

0.7716) 

R 

0  C  F  I  T  (JUNE  1993 

1  VERSION) 

: 

MAXI 

MUM 

L 

I  K 

E 

L 

IHOOD  EST 

I 

M  A  T  I  0 

N 

0 

F  A 

B 

I  N 

0 

R 

M  A  L  ROC  C 

U 

R  V  E 

F  R 

0 

M 

R 

A 

TING  DATA 

DATA  DESCRIPTION:  Reader  10,  FAS /AD 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT’ 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  23. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  96.  2. 

CASES  20.  2. 


3  4  5 

1.  1.  0. 

0.  0.  1. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0100  0.0200  0.0400  1.0000 
TPF:  0.0000  0.0435  0.0435  0.0435  0.1304  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  0.0057  B=  0.6871 

Z(K)=  1.7511  2.0542  2.3268  2.5762 

LOGL=  -34.0185 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.1095  B=  0.7126 

Z(K)=  1.7463  2.1891  2.3811  2.7516 

LOGL=  -33.5911 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


A 

B 

Z(  1) 
Z(  2) 


VARIANCE-COVARIANCE  MATRIX: 

1.2379  0.5909  0.0566  -0.0612  -0.1250  -0.2807 

0.5909  0.3157  0.0114  -0.0532  -0.0884  -0.1745 

0.0566  0.0114  0.0514  0.0437  0.0408  0.0356 

-0.0612  -0.0532  0.0437  0.0968  0.0986  0.1071 


116 


Z(  3)  -0.1250  -0.0884  0.0408  0.0986  0.1412 
Z(  4)  -0.2807  -0.1745  0.0356  0.1071  0.1559 


0.1559 

0.3118 


A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.9453 

0.9453  1.0000 

0.2243  0.0898 

-0.1767  -0.3045 
-0.2989  -0.4188 
-0.4518  -0.5560 


0.2243  -0.1767 
0.0898  -0.3045 
1.0000  0.6202 
0.6202  1.0000 
0.4794  0.8436 
0.2813  0.6162 


-0.2989  -0.4518 
-0.4188  -0.5560 
0.4794  0.2813 

0.8436  0.6162 

1.0000  0.7429 

0.7429  1.0000 


AREA  =  0.5355  STD.  DEV. (AREA)  -  0.3512 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.0421 

( 

0.0027 

r 

0.2502 

) 

0.010 

0.0607 

( 

0.0078 

r 

0.2487 

) 

0.020 

0.0878 

( 

0.0181 

f 

0.2692 

) 

0.030 

0.1091 

( 

0.0260 

¥ 

0.3017 

) 

0.040 

0.1275 

( 

0.0313 

f 

0.3392 

) 

0.050 

0.1439 

( 

0.0346 

¥ 

0.3787 

) 

0.060 

0,1590 

( 

0.0366 

¥ 

0.4182 

) 

0.070 

0.1730 

( 

0.0378 

¥ 

0.4570 

) 

0.080 

0.1862 

( 

0.0384 

¥ 

0.4942 

) 

0.090 

0.1987 

( 

0.0386 

¥ 

0.5298 

) 

0.100 

0.2107 

( 

0.0386 

¥ 

0.5634 

) 

0.110 

0.2222 

( 

0,0384 

¥ 

0.5951 

) 

0.120 

0.2333 

( 

0.0380 

¥ 

0.6249 

) 

0.130 

0.2441 

( 

0.0376 

¥ 

0.6528 

) 

0.140 

0.2545 

( 

0.0371 

¥ 

0.6789 

) 

0.150 

0,2646 

( 

0.0366 

¥ 

0.7033 

) 

0.200 

0.3120 

( 

0.0337 

¥ 

0.8021 

) 

0.250 

0.3553 

( 

0.0308 

¥ 

0.8703 

) 

0.300 

0.3959 

( 

0.0281 

¥ 

0.9165 

) 

0.400 

0.4718 

( 

0.0233 

¥ 

0.9678 

) 

0.500 

0.5436 

( 

0.0192 

¥ 

0.9890 

) 

0.600 

0.6140 

( 

0.0155 

¥ 

0.9969 

) 

0.700 

0,6854 

( 

0.0122 

¥ 

0.9993 

) 

0.800 

0.7609 

( 

0.0091 

¥ 

0.9999 

) 

0.900 

0.8468 

( 

0.0058 

¥ 

1.0000 

) 

0.950 

0.9001 

( 

0.0040 

¥ 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0030, 

0.0321) 

(0.0001, 

0.0043) 

(0.0487, 

0.1420) 

(0.0086, 

0.0562) 

(0.0009, 

0.0173) 

(0.0500, 

0.1440) 

(0.0143, 

0.0735) 

(0.0026, 

0.0297) 

(0.0571, 

0.1548) 

(0.0404, 

0.1282) 

(0.0142, 

0.0733) 

(0.0965, 

0.2066) 
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R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  10,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALITY 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  100. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  91.  6. 

CASES  37 .  22 . 


3  4  5 

2.  1.  0. 

20.  6.  15. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0100  0.0300  0.0900  1.0000 
TPF:  0.0000  0.1500  0.2100  0.4100  0.6300  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.8597  B=  1.1295 

Z(K)=  1.3410  1.8812  2.3268  2.5762 

LOGL=  -186.6256 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.6595  B=  0.9936 

Z(K)=  1.3387  1.8972  2.4656  2.7230 

LOGL=  -186.2753 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


VARIANCE-COVARIANCE  MATRIX : 


A 

0.3016 

0.1713 

0.0444 

-0.0289 

-0.1175 

-0.1599 

B 

0.1713 

0.1138 

0.0102 

-0.0408 

-0.1014 

-0.1302 

Z( 

1) 

0.0444 

0.0102 

0.0310 

0.0251 

0.0193 

0.0167 

Z( 

2) 

-0.0289 

-0.0408 

0.0251 

0.0514 

0.0715 

0.0813 

Z( 

3) 

-0.1175 

-0.1014 

0.0193 

0.0715 

0.1364 

0.1605 

Z( 

4) 

-0.1599 

-0.1302 

0.0167 

0.0813 

0.1605 

0.2006 

A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 


1.0000  0.9249 
0.9249  1.0000 
0.4596  0.1723 
0.2320  -0.5331 
0.5793  -0.8142 
0.6499  -0.8616 


0.4596  -0.2320 
0.1723  -0.5331 
1.0000  0.6282 
0.6282  1.0000 
0.2976  0.8537 
0.2124  0.8011 


-0.5793  -0.6499 
-0.8142  -0.8616 
0.2976  0.2124 

0.8537  0.8011 

1.0000  0.9706 

0.9706  1.0000 


AREA  =  0.8804  STD.  DEV. (AREA)  =  0.0438 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 
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FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1840 

( 

0.0429 

f 

0.4670 

) 

0.010 

0.2571 

( 

0.0914 

/ 

0.5109 

) 

0.020 

0.3514 

( 

0.1766 

r 

0.5656 

) 

0.030 

0.4169 

( 

0.2456 

r 

0.6060 

) 

0.040 

0.4680 

( 

0.3013 

r 

0.6405 

) 

0.050 

0.5099 

( 

0.3465 

r 

0.6716 

) 

0.060 

0.5455 

( 

0.3835 

r 

0.7002 

) 

0.070 

0.5765 

( 

0.4141 

r 

0.7266 

) 

0,080 

0.6038 

( 

0.4398 

9 

0.7511 

) 

0.090 

0.6282 

( 

0.4616 

9 

0.7736 

) 

0,100 

0.6502 

( 

0.4803 

9 

0.7943 

) 

0.110 

0.6703 

( 

0.4967 

9 

0.8132 

) 

0.120 

0.6886 

( 

0.5111 

9 

0.8305 

) 

0.130 

0,7055 

( 

0.5240 

9 

0.8462 

) 

0.140 

0.7211 

( 

0.5356 

9 

0.8605 

) 

0.150 

0.7356 

( 

0.5462 

9 

0.8736 

) 

0.200 

0.7949 

( 

0.5882 

9 

0,9228 

) 

0.250 

0.8388 

( 

0.6195 

9 

0.9530 

) 

0,300 

0.8726 

( 

0.6448 

9 

0.9717 

) 

0.400 

0.9205 

( 

0.6859 

9 

0.9902 

) 

0.500 

0.9515 

( 

0.7201 

9 

0.9969 

) 

0.600 

0.9720 

( 

0.7511 

9 

0.9992 

) 

0.700 

0.9854 

( 

0.7815 

9 

0.9998 

) 

0.800 

0.9937 

( 

0.8134 

9 

1.0000 

) 

0.900 

0.9983 

( 

0.8520 

9 

1.0000 

) 

0.950 

0.9995 

( 

0.8792 

9 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(0.0032, 

0.1477) 

(0.0002, 

0.0275) 

(0.0325, 

0.4310) 

(0.0068, 

0.2146) 

(0.0007, 

0.0656) 

(0.0408, 

0.4716) 

o 

o 

00 

0.4108) 

(0.0096, 

0.2524) 

(0.0731, 

0.5855) 

(0.0903, 

0.6291) 

(0.0461, 

0.4947) 

(0.1602, 

0.7492) 
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R  0  C  F  I  T  (JUNE  1993  VERSION)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING,  DATA 


DATA  DESCRIPTION:  Reader  11,  Mass  Question 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  42. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0600  0.0700  0.2400  1.0000 
TPF:  0.0000  0.6190  0.6667  0.6905  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  0.6234  B=  0.1714 
Z(K)=  0.7060  1.4761  1.5551 

LOGL=  -113.0300 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.6617  B=  0.2222 

Z(K)=  0.7099  1.4142  1.5978 

LOGL=  -111.9829 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


VARIANCE-COVARIANCE  MATRIX: 

A  0.0626  0.0207  0.0054  -0.0002  -0.0026 
B  0.0207  0.0174  0.0017  -0.0046  -0.0074 
Z(  1)  0.0054  0.0017  0.0189  0.0132  0.0121 
Z(  2)  -0.0002  -0.0046  0.0132  0.0329  0.0316 
Z(  3)  -0.0026  -0.0074  0.0121  0.0316  0.0415 


CORRELATION  MATRIX: 

A  1.0000  0.6284  0.1570  -0.0048  -0.0516 
B  0.6284  1.0000  0.0938  -0.1911  -0.2756 
Z(  1)  0.1570  0.0938  1.0000  0.5295  0.4318 
Z(  2)  -0.0048  -0.1911  0.5295  1.0000  0.8557 
Z(  3)  -0.0516  -0.2756  0.4318  0.8557  1.0000 


AREA  =  0.7408  STD.  DEV. (AREA)  =  0.0756 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


1  2  3  4  5 

76.  17.  0.  1.  6. 

13.  1.  0.  2.  26. 
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0.005 

0.5356 

(  0.3324 

/ 

0.7297 

) 

0.010 

0.5575 

(  0.3684 

r 

0.7342 

) 

0.020 

0.5813 

(  0.4066 

r 

0.7411 

) 

0.030 

0.5963 

(  0.4297 

r 

0.7468 

) 

0.040 

0.6074 

(  0.4462 

r 

0.7519 

) 

0.050 

0.6164 

(  0.4590 

t 

0.7565 

) 

0.060 

0.6240 

(  0.4694 

r 

0.7608 

) 

0.070 

0.6307 

(  0.4782 

r 

0.7649 

) 

0.080 

0.6366 

(  0.4856 

r 

0.7688 

) 

0.090 

0.6420 

(  0.4921 

r 

0.7725 

) 

0.100 

0.6469 

(  0.4979 

f 

0.7761 

) 

0.110 

0.6514 

(  0.5030 

r 

0.7796 

) 

0.120 

0.6556 

(  0.5076 

f 

0.7829 

) 

0.130 

0.6596 

(  0.5117 

f 

0.7862 

) 

0.140 

0.6633 

(  0.5155 

r 

0.7894 

) 

0.150 

0.6669 

(  0.5190 

r 

0.7925 

) 

0.200 

0.6825 

(  0.5329 

f 

0.8070 

) 

0.250 

0.6956 

(  0.5428 

f 

0.8203 

) 

0.300 

0.7072 

(  0.5503 

r 

0.8325 

) 

0.400 

0.7276 

(  0.5609 

r 

0.8549 

) 

0.500 

0.7459 

(  0.5679 

r 

0.8754 

) 

0.600 

0.7636 

(  0.5729 

f 

0.8947 

) 

0.700 

0.7817 

(  0.5765 

r 

0.9136 

) 

0.800 

0.8020 

(  0.5789 

r 

0.9329 

) 

0.900 

0.8280 

{  0.5800 

r 

0.9546 

) 

0.950 

0.8478 

(  0.5795 

r 

0.9681 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 


LOWER  BOUND  UPPER  BOUND 

FPF  ,  TPF  )  (  FPF  ,  TPF  ) 


(0.0550,  0.6204) 
(0.0786,  0.6359) 
(0.2389,  0.6928) 


(0.0229,  0.5862)  (0.1154, 

(0.0384,  0.6058)  (0.1448, 

(0.1637,  0.6715)  (0.3298, 

R  0  C  F  I  T  (JUNE  1993  VERSION) 


0.6537) 

0.6651) 

0.7136) 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  11,  MicroCalcifications 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 
NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  51. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  95 .  0 . 

CASES  26.  2. 


3  4  5 

0.  1.  4. 

0.  0.  23. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0400  0.0500  0.0500  1.0000 
TPF:  0.0000  0.4510  0.4510  0.4902  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 
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A=  1.5769  B=  0.9713 

Z(K)=  1.5452  1.6452  1.7511 

LOGL=  -67.9594 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  I 


PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.6025  B=  0.9900 

Z(K)=  1.6444  1.7102  1.7441 

LOGL=  -66.6294 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  1 


A 

VARIANCE-COVARIANCE  MATRIX : 
4.1910  2.4630  0.0735  -0.0363 

-0.0944 

B 

2.4630 

1.4733 

0.0178 

-0.0482 

-0.0832 

Z( 

1) 

0.0735 

0.0178 

0.0446 

0.0434 

0.0428 

Z( 

2) 

-0.0363 

-0.0482 

0.0434 

0.0474 

0.0483 

Z( 

3) 

-0.0944 

-0.0832 

0.0428 

0.0483 

0.0512 

A 

CORRELATION  MATRIX: 
1.0000  0.9912  0.1699 

-0.0814 

-0.2037 

B 

0.9912 

1.0000 

0.0695 

-0.1825 

-0.3028 

Z( 

1) 

0.1699 

0.0695 

1.0000 

0.9449 

0.8960 

Z( 

2) 

-0.0814 

-0.1825 

0.9449 

1.0000 

0.9806 

Z( 

3) 

-0.2037 

-0.3028 

0.8960 

0.9806 

1.0000 

AREA  =  0.8726  STD.  DEV. (AREA)  =  0.1617 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1716 

( 

0.0008 

r 

0.8977 

) 

0.010 

0.2416 

( 

0.0095 

r 

0.8277 

) 

0.020 

0.3332 

( 

0.0688 

f 

0.7331 

) 

0.030 

0.3975 

( 

0,1615 

r 

0.6802 

) 

0.040 

0.4478 

( 

0.2436 

r 

0.6673 

) 

0.050 

0.4895 

( 

0.2873 

r 

0.6945 

) 

0.060 

0.5251 

( 

0.2958 

f 

0.7461 

) 

0.070 

0.5561 

( 

0.2860 

r 

0.8016 

) 

0.080 

0.5836 

( 

0.2690 

r 

0.8504 

) 

0.090 

0.6083 

( 

0.2500 

r 

0.8896 

) 

0.100 

0.6306 

( 

0.2310 

f 

0.9197 

) 

0.110 

0.6510 

( 

0.2128 

r 

0.9421 

) 

0.120 

0.6697 

( 

0.1959 

r 

0.9586 

) 

0.130 

0.6870 

( 

0.1802 

f 

0.9706 

) 

0.140 

0.7030 

( 

0.1658 

r 

0.9792 

) 

0.150 

0.7178 

( 

0.1525 

f 

0.9853 

) 

0.200 

0.7792 

( 

0.1010 

r 

0.9976 

) 

0.250 

0.8251 

( 

0.0672 

r 

0.9996 

) 

0.300 

0.8608 

( 

0.0448 

r 

0.9999 

) 

0.400 

0.9118 

( 

0.0195 

f 

1.0000 

) 

0.500 

0.9455 

( 

0.0080 

r 

1.0000 

) 

0.600 

0.9681 

( 

0.0029 

r 

1.0000 

) 

0.700 

0.9831 

( 

0.0009 

r 

1.0000 

) 

0.800 

0.9926 

( 

0.0002 

r 

1.0000 

) 
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0.900  0.9980 
0.950  0.9994 


(  0.0000 

(  0.0000 


1.0000  ) 

1.0000  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER  BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  ,  TPF  ) 

(  FPF  , 

TPF  ) 

(0.0406, 

0.4506) 

(0.0143,  0.2866) 

(0.0967, 

0.6236) 

(0.0436, 

0.4639) 

(0.0163,  0.3040) 

(0.0996, 

0.6299) 

(0.0500, 

0.4898) 

(0.0198,  0.3316) 

(0.1093, 

0.6497) 

R 

0  C  F  I  T  (JUNE  1993 

1  VERSION) 

1 

MAXI 

MUM 

L 

I  K 

E 

L 

IHOOD  EST 

I 

M  A  T  I  0 

N 

0 

F  A 

B 

I  N 

0 

R 

M  A  L  ROC  C 

U 

R  V  E 

F  R 

0 

M 

R 

A 

TING  DATA 

DATA  DESCRIPTION:  Reader  11,  FAS /AD 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT’ 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100. 


NO.  OF  ACTUALLY  POSITIVE  CASES  =  23. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  73.  1. 

CASES  13 .  1 . 


3  4  5 

0.  14.  12. 

0.  5.  4. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.1200  0.2600  0.2700  1.0000 
TPF:  0.0000  0.1739  0.3913  0.4348  1.0000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  0.6094  B=  1.3192 

Z(K)=  0.6125  0.6430  1.1751 

LOGL=  -106.1220 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE 


PROCEDURE  CONVERGES  AFTER  3  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.6843  B=  1.3990 

Z(K)=  0.6117  0.6559  1.1707 

LOGL=  -105.9773 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  ! 


VARIANCE-COVARIANCE  MATRIX : 


A 

0.3212 

0.2899 

0.0325 

0.0300 

-0.0014 

B 

0.2899 

0.3677 

0.0120 

0.0086 

-0.0324 

Z( 

1) 

0.0325 

0.0120 

0.0180 

0.0176 

0.0132 

Z( 

2) 

0.0300 

0.0086 

0.0176 

0.0181 

0.0140 

Z( 

3) 

-0.0014 

-0.0324 

0.0132 

0.0140 

0.0263 
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COREIELATION  MATRIX: 


A 

1.0000 

0.8435 

0.4273 

0.3936 

-0.0149 

B 

0.8435 

1.0000 

0.1470 

0.1056 

-0.3302 

Z( 

1) 

0.4273 

0.1470 

1.0000 

0.9726 

0.6091 

Z( 

2) 

0.3936 

0.1056 

0.9726 

1.0000 

0.6421 

Z( 

3) 

-0.0149 

-0.3302 

0.6091 

0.6421 

1.0000 

AREA-  0.6547  STD.  DEV. (AREA)  =  0.0889 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.0018 

( 

0.0000 

r 

0.2380 

) 

0.010 

0.0051 

( 

0.0000 

r 

0.2586 

) 

0.020 

0.0143 

( 

0.0001 

r 

0.2840 

) 

0.030 

0.0257 

( 

0.0004 

f 

0.3022 

) 

0.040 

0.0387 

( 

0.0011 

f 

0.3174 

) 

0.050 

0.0529 

( 

0.0026 

r 

0.3310 

) 

0.060 

0.0679 

( 

0.0049 

r 

0.3438 

) 

0.070 

0.0837 

( 

0.0084 

r 

0.3561 

) 

0.080 

0.1000 

( 

0.0130 

f 

0.3683 

) 

0.090 

0.1167 

( 

0.0188 

r 

0.3804 

) 

0.100 

0.1337 

( 

0.0259 

f 

0.3927 

) 

0.110 

0.1511 

( 

0.0341 

r 

0.4053 

) 

0.120 

0.1686 

( 

0.0434 

f 

0.4182 

) 

0.130 

0.1863 

( 

0.0536 

r 

0.4316 

) 

0.140 

0.2041 

( 

0.0646 

r 

0.4456 

) 

0.150 

0.2219 

( 

0.0762 

f 

0.4601 

) 

0.200 

0.3110 

( 

0.1372 

f 

0.5426 

) 

0.250 

0.3979 

( 

0.1919 

r 

0.6380 

) 

0.300 

0.4805 

( 

0.2348 

r 

0.7342 

) 

0.400 

0.6295 

( 

0.2939 

r 

0.8855 

) 

0.500 

0.7531 

( 

0.3348 

r 

0.9637 

) 

0.600 

0.8504 

( 

0.3685 

r 

0.9921 

) 

0.700 

0.9218 

( 

0.4003 

r 

0.9990 

) 

0.800 

0.9687 

( 

0.4343 

t 

0.9999 

) 

0.900 

0.9934 

( 

0.4784 

r 

1.0000 

) 

0.950 

0.9986 

( 

0.5130 

r 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER  BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  ,  TPF  ) 

(  FPF  , 

TPF  ) 

(0.1209, 

0.1702) 

(0.0683,  0.0811) 

(0.1968, 

0.3053) 

(0.2559, 

0.4078) 

(0.1789,  0.2735) 

(0.3475, 

0.5540) 

(0.2704, 

0.4319) 

(0.1908,  0.2948) 

(0.3637, 

0.5779) 

R 

0  C  F  I  T  (JUNE 

1993  VERSION) 

: 

MAXI 

MUM 

L 

I 
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E 

L 

I  H  0  0  D  E  S 
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0 
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0 
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A 
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DATA  DESCRIPTION:  Reader  11,  Benign  or  Malignant 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY  (E.G.,  THAT  ABNORMALIT' 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  100.  NO.  OF  ACTUALLY  POSITIVE  CASES  =  100. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 
CASES  56.  31. 

CASES  21.  38. 


3  4  5 

13.  0.  0. 

18.  14.  9. 


OBSERVED  OPERATING  POINTS: 

FPF:  0.0000  0.0000  0.0000  0.1300  0.4400  1.0000 
TPF:  0.0000  0.0900  0.2300  0.4100  0.7900  1.0000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  0.8157  B=  0.7526 

Z(K)=  0.1507  1.1265  2.4762  2.5762 

LOGL=  -270.4485 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  0.8626  B=  0.7836 

Z(K)=  0.1283  1.2717  2.1074  2.8588 

LOGL=  -249.0090 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED  BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  : 


VARIANCE-COVARIANCE  MATRIX: 


A 

0.0316 

0.0134 

0.0127 

0.0070 

-0.0015 

-0.0119 

B 

0.0134 

0.0174 

0.0044 

-0.0078 

-0.0229 

-0.0396 

Z( 

1) 

0.0127 

0.0044 

0.0157 

0.0086 

0.0045 

0.0006 

Z( 

2) 

0.0070 

-0.0078 

0.0086 

0.0247 

0.0289 

0.0356 

Z  ( 

3) 

-0.0015 

-0.0229 

0.0045 

0.0289 

0.0647 

0.0826 

Z( 

4) 

“0.0119 

-0.0396 

0.0006 

0.0356 

0.0826 

0.1506 

A 

CORRELATION  MATRIX: 
1.0000  0.5699  0.5690 

0.2500 

-0.0329 

-0.1725 

B 

0.5699 

1.0000 

0.2638 

-0.3768 

-0.6815 

-0.7725 

Z( 

1) 

0.5690 

0.2638 

1.0000 

0.4374 

0.1427 

0.0120 

Z  ( 

2) 

0.2500 

-0.3768 

0.4374 

1.0000 

0.7221 

0.5829 

Z( 

3) 

-0.0329 

-0.6815 

0.1427 

0.7221 

1.0000 

0.8373 

Z( 

4) 

-0.1725 

-0.7725 

0.0120 

0.5829 

0.8373 

1.0000 

AREA  = 

0.7514 

STD 

.  DEV. (AREA)  = 

0.0382 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

0.005 

0.1238 

(  0.0441  , 

0.2718  ) 

0.010 

0.1684 

(  0.0728  , 

0.3207  ) 

0.020 

0.2275 

(  0.1178  , 

0.3791  ) 

0.030 

0.2704 

(  0.1544  , 

0.4187  ) 
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0.040 

0.3052 

( 

0.1860 

r 

0.4497 

) 

0.050 

0.3349 

( 

0.2142 

t 

0.4756 

) 

0.060 

0.3609 

( 

0.2397 

r 

0.4982 

) 

0.070 

0.3844 

( 

0.2631 

r 

0.5183 

) 

0.080 

0.4057 

( 

0.2847 

r 

0.5366 

) 

0.090 

0.4254 

( 

0.3048 

r 

0.5534 

) 

0.100 

0.4436 

( 

0.3237 

r 

0.5690 

) 

0.110 

0.4607 

( 

0.3415 

t 

0.5837 

) 

0.120 

0.4768 

( 

0.3582 

f 

0.5975 

) 

0.130 

0.4920 

( 

0.3741 

r 

0.6106 

) 

0.140 

0.5064 

( 

0.3892 

r 

0.6230 

) 

0.150 

0.5201 

( 

0.4036 

r 

0.6350 

) 

0.200 

0.5805 

( 

0.4667 

r 

0.6880 

) 

0.250 

0.6309 

( 

0.5188 

t 

0.7329 

) 

0.300 

0.6744 

( 

0.5632 

r 

0.7719 

) 

0.400 

0.7468 

( 

0.6364 

f 

0.8364 

) 

0.500 

0.8058 

( 

0.6965 

f 

0.8870 

) 

0.600 

0.8556 

( 

0.7491 

r 

0.9265 

) 

0.700 

0.8985 

( 

0.7977 

r 

0.9566 

) 

0.800 

0.9360 

( 

0.8454 

r 

0.9787 

) 

0.900 

0.9690 

( 

0.8970 

f 

0.9932 

) 

0.950 

0.9843 

( 

0.9286 

r 

0.9977 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVALS  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  fpf  , 

TPF  ) 

(  fpf  , 

TPF  ) 

1 — 1 
CM 
O 
O 

O 

0.0842) 

(0.0001, 

0.0242) 

(0.0179, 

0.2172) 

(0.0175, 

0.2151) 

CO 

o 

o 

o 

0.1191) 

(0.0538, 

0.3453) 

(0.1017, 

0.4468) 

(0.0571, 

0.3537) 

(0.1676, 

0.5428) 

(0.4490, 

0.7770) 

(0.3543, 

0.7155) 

(0.5467, 

0.8301) 
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UNIVERSITY  OF  WASHINGTON 

SEATTLE,  WASHINGTON  98195 


Brent  K.  Stewart,  Ph.D. 
Associate  Professor  and  Director 
Diagnostic  Physics  Laboratory 
Department  of  Radiology,  RC-05 


Samuel  J.  Dwyer  HI,  Ph.D. 

Professor 

Department  of  Radiology 
University  of  Virginia 
MR-4  Room  1190 
Charlottesville,  VA  22908 

Dear  Sam; 

Please  find  enclosed  the  findings  of  my  visit  to  the  Medical  College  of  Virginia  on 
1/27/95  and  the  University  of  Virginia  on  1/28/95  as  consultant  on  the  US  Army 
Medical  Research  and  Development  Command  grant  entitled;  "Evaluation  of  a  Digital 
Telemammography  System;  a  Model  for  a  Regional  System." 

Visit  to  the  Medical  College  of  Virginia 

On  1/27/95, 1  met  with  Ellen  Shaw  de  Parades,  M.D.,  Chief  of  Mammography  at  the 
Medical  College  of  Virginia's  Department  of  Radiology  and  Principal  Investigator  of  the 
telemammography  grant.  The  purpose  of  the  consulting  at  the  Medical  College  of 
Virginia  was  to  analyze  the  design  of  the  Receiver  Operator  Characteristics  (ROC) 
studies,  comment  on  the  method  of  selecting  images  for  the  study,  examine  the  images 
already  collected  for  the  study,  and  discuss  strategies  for  analysis  of  the  ROC  after  the 
testing  has  concluded.  I  prepared  a  list  of  questions  (given  below).  I  also  sat  through  a 
few  of  the  tests  to  give  advice  on  reading  room  environment,  e.g.,  view  box  luminance 
and  glare. 

I  submitted  a  list  of  questions  to  Dr.  de  Parades  regarding  the  analog  film  and  digital 
softcopy  ROC  testing: 

1.  Has  the  ROC  testing  commenced  and  if  so,  how  far  along  is  it? 

2.  Has  the  ROC  study  design  changed  significantly  from  that  stated  in  the  initial 
proposal? 

3.  How  are  the  mammograms  for  the  ROC  study  selected? 

4.  You  are  selecting  age-matched  normal  controls.  Are  you  matching  these  normal 
mammograms  for  overall  parenchymal  density  as  well?  If  so,  how  are  you 
accomplishing  this? 

5.  Are  the  initial  200  mammograms  cited  in  the  grant  application  digitized  yet?  If  so, 
what  is  keeping  you  from  initiating  the  digital  softcopy  ROC  portion  of  the  study? 


NW040J  Health  Sciences  Bldg. 
206.548.6252  (office) 
206.543.3495  (fax) 
bstewart@u.washington.edu 
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6.  For  mammogram  digitization,  what  quality  control/ assurance  program  have  you 
instituted? 

7.  In  the  original  grant  application,  it  was  stated  that  the  digital  softcopy  review  might 
occur  on  any  of  ten  different  2K  resolution  workstations  throughout  the  UVA 
Department  of  Radiology.  Unless  these  workstation's  monitors  are  periodically  and 
effectively  calibrated,  this  might  confound  the  ROC  results. 

8.  In  the  original  grant  application,  the  images  read  at  the  remote  Northridge  outpatient 
clinic  were  to  be  subjected  to  a  preference  test  (scale:  1-5).  Would  it  be  better  to  have 
these  cases  overread  by  mammographers  at  UVA  and  statistically  calculate  the  analysis 
of  variance? 

9.  Reading  all  of  the  analog  images  first  produces  a  bias  in  the  ROC  test.  It  would  be 
better  to  have  one-half  of  the  radiologists  read  the  digital  softcopy  images  first  and  the 
other  radiologists  read  the  analog  images  first.  Of  course,  as  there  will  be  multiple 
reading  sessions  for  each  modality,  each  session  could  be  randomly  picked  from  analog 
or  digital.  This  bias  may,  of  course,  be  confounded  as  the  radiologists  will  know  which 
images  are  analog  and  which  are  digital. 

10.  Is  ground  truth  available  for  all  of  the  films  to  be  used  in  the  study?  What  aside 
information  are  you  using  to  establish  ground  truth?  Is  there  a  truth  committee?  If  so, 
who  is  on  it  and  how  do  they  arrive  a  conclusion  regarding  a  case  without  unanimity? 

11.  A  random  number  generator  should  also  be  used  for  ordering  the  analog  and  digital 
normals  and  abnormals  in  each  of  the  ROC  study  sessions.  Is  this  the  case  and  if  not, 
why  not? 

12.  Who  will  be  collecting,  collating  and  performing  the  analysis  of  the  ROC  test  result 
data?  Will  you  be  using  one  of  the  standard  software  packages  like  ROCFIT  or 
CORROC  from  the  University  of  Chicago? 

13.  In  the  grant,  it  is  stated  that  in  addition  to  the  50|im  digitized  radiographs,  that  some 
would  be  digitized  at  23|un.  If  so,  how  many  and  are  you  adding  this  as  another  section 
of  the  original  ROC  study? 

14.  It  appears  from  the  grant  application  that  the  ROC  results  will  be  pooled  for  the  four 
different  pathology  types.  Is  this  still  the  case?  Will  you  achieve  sufficient  statistical 
power  in  the  non-pooled  case? 

15.  How  has  splitting  the  grant  across  institutions  (UVA  and  MCV)  affected  the  design 
and  execution  of  the  proposed  work? 

16.  Will  you  be  using  the  BIRADS  system  information  for  patient  selection?  It  doesn't 
appear  that  the  RadCare  radiology  information  system  in  place  at  the  MCV  will 
facilitate  on-line  image  selection  for  the  ROC.  What  system  will  you  have  to  help 
automate  patient  selection? 

17.  Are  there  any  problems  in  selecting  cases  from  both  MCV  and  UVA  in  terms  of 
image  quality  differences?  There  should  be  differences  in  film  type,  screen  type, 
mammography  machine  output,  film  processing,  etc. 

18.  It  will  be  possible  to  time  the  readers  using  the  computer  in  the  softcopy  display 
workstation.  Are  you  planning  on  doing  this?  If  so,  could  the  radiologist  write-down 
the  start  and  end  times  on  the  scoring  sheets? 
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Advice  on  ROC  Reading  Sessions: 

I  also  sat  through  three  sessions  of  analog  ROC  testing  with  one  private  practice 
mammographer  and  two  MCV  faculty  radiologists.  A  specific  mammography  view 
panel  was  used.  This  viewing  panel  had  the  capability  of  shuttering  out  extraneous 
light  around  the  edges  of  the  films,  however,  this  was  not  done  in  all  cases  by  all 
radiologists.  Both  8"xl0"  and  10"xl2"  films  were  used.  One  row  was  used  at  a  time. 
The  medio-lateral  views  paired  on  the  left,  the  cranial-caudal  radiographs  were  paired 
on  the  right.  The  room,  the  mammography  reading  room,  was  fairly  quiet,  but  was 
simultaneously  used  by  another  radiologist  and  a  resident,  as  well  as  Dr.  de  Parades 
during  the  ROC  sessions.  The  readers  did  interrupt  their  reading  sessions  to  speak  with 
colleagues  or  answer  the  phone /pager.  There  were  no  overhead  lights  to  contend  with 
and  there  was  no  light  reflections  on  the  ROC  viewbox. 

A  magnifying  lens  was  provided  (will  an  analogous  "zooming"  capability  be  added  to 
the  softcopy  display  workstation  as  well?).  A  hot  lamp  was  available  (will  an  analogous 
grayscale  look-up  table  facility  be  added  to  the  softcopy  display  workstation  as  well?). 

The  reading  sessions  consist  of  50  patient  studies,  each  consisting  of  four  radiographs  (2 
CC/2MLO).  There  is  one  three  ring  binder  notebook  for  each  reader.  All  of  the 
instructions  for  each  reader  are  in  the  notebooks,  as  well  as  all  of  the  reader  responses 
for  each  patient  case  read. 

On  the  average,  the  magnifying  glass  was  used  in  96%  of  the  cases  read,  whereas,  the 
hot  lamp  was  used  only  sparingly,  about  10%  of  the  time.  The  radiologists  always 
started  with  the  MLO  views  and  then  the  CC  views.  Any  zooming  and  panning  would 
need  to  happen  quickly  to  be  effective  (not  slowing  down  the  reading  process 
significantly.  There  were  several  instances  of  the  radiologists  being  interrupted  for 
pages  and  consultations.  If  a  timer  were  to  be  integrated  into  the  softcopy  reading 
workstation,  a  pause  button  would  be  useful. 

There  were  several  instances  where  films  were  displaced  vertically  to  come  into 
registration  (vertical  shift).  This  capability  may  need  to  be  added  to  the  digital  review 
workstation.  It  would  be  very  hard  to  be  the  video  monitors  close  enough  for  bi-lateral 
comparison.  Digital  panning  may  be  necessary.  On  the  average  it  took  two  minutes 
and  18  seconds  to  read  one  of  the  50  studies  in  the  ROC  study  list. 

Visit  to  the  University  of  Virginia 

On  1/28/95, 1  met  with  Samuel  J.  Dwyer,  Ph.D.,  Director  of  PACS  and  Co-Prinicpal 
Investigator  of  the  telemammography  grant  at  the  Medical  College  of  Virginia's 
Department  of  Radiology.  I  also  met  with  Beth  Elias,  B.S.,  the  systems  analyst  for  the 
telemammography  grant.  The  purpose  of  the  consulting  at  the  University  of  Virginia 
was  to  examine  and  provide  recommendations  for  mammogram  digitization,  image 
presentation  on  the  viewing  monitors,  and  image  processing  functionality. 
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I  made  several  recommendations  regarding  image  digitization  quality  control, 
specifically  daily  digitization  of  a  standard  test  pattern  and  periodic  cleaning  and 
calibration  of  the  digitizer.  I  also  suggested  several  means  of  displaying  the  image 
digitally  to  the  radiologists  for  that  portion  of  the  ROC  testing.  There  were  also 
questions  regarding  where  an  additional  image  reading  station  for  the  MCV  portion  of 
the  digital  ROC  testing  were  coming  from.  It  might  be  the  case  the  E-systems  will  loan 
as  system  to  the  MCV  for  the  duration  of  the  ROC  testing.  Due  to  construction  and  a 
snow  storm,  it  was  not  convenient  to  visit  the  Northridge  site. 

Image  Digitization: 

The  images  are  being  digitized  at  the  UVA  under  the  direction  of  Ms.  Elias.  A  Lumisys 
digitizer,  model  150,  is  being  used  for  the  digitization.  A  SMPTE  (Society  of  Motion 
Picture  Test  Engineers)  is  being  used  for  daily  grayscale  and  resolution  quality  control. 
The  mirrors  of  the  system  are  cleaned  bi-monthly.  Every  four  months,  a  field  engineer 
from  E-systems  recalibrated  the  digitizer  densitometry. 

It  was  suggested  that  the  name  of  the  patient,  the  patient  identification  number,  the  date 
of  the  examination  and  the  name  of  the  institution  be  masked  off  with  electrical  tape 
prior  to  digitization.  It  was  also  suggested  that  a  single  normal  mammogram  be  used 
for  daily  grayscale  quality  control.  This  mammogram  could  be  digitized  every  day, 
prior  to  digitization  of  mammograms  for  the  digital  part  of  the  study.  Once  registered 
spatially,  the  daily  mammogram  could  be  digitally  subtracted  from  the  baseline  one  and 
the  difference  image  studied.  If  it  appeared  that  there  is  more  than  simple  noise 
differences  in  the  difference  image,  e.g.,  structure  evident,  then  the  densitometry  might 
need  to  be  adjusted  more  often  than  every  four  months. 

Image  Presentation  on  the  Viewing  Monitors: 

How  many  monitors  are  going  to  be  used  for  the  workstations  in  the  study?  Only  two. 
It  was  observed  above  that  the  radiologists  reviewing  the  analog  cases  switched  back 
and  forth  between  the  CC  and  MLO  pairs  quite  often.  If  only  two  video  monitors  were 
used,  this  would  severely  hamper  both  the  comparison  necessary  for  diagnosis,  but 
significantly  increase  the  interpretation  time  as  well.  Methods  were  discussed  with  Ms. 
Elias  for  quickly  context  switching  between  the  two  sets  (MLO  and  CC)  of 
mammograms  for  each  patient.  The  limiting  factor  here  is  that  it  is  only  possible  to  load 
two  mammograms  into  the  E-systems  MegaScan  2K  monitor  digital  frame  buffer  (32 
Mbyte  limit).  Having  to  re-paint  the  frame  buffers  from  magnetic  disk  for  each  MLO  <- 
>  CC  context  switch  will  most  likely  be  interminably  slow. 

It  was  also  suggested  that  a  sequential  worklist  of  patients  for  the  softcopy  review 
workstation  portion  of  the  ROC  study  be  instituted.  Currently,  the  radiologist  has  to 
select  images  from  a  pull-down  menu  list  with  small  font.  The  easiest  thing  for  the 
radiologists  to  have  to  do  would  be  to  push  a  "hot  key"  to  advance  to  the  next  patient  in 
the  ROC  study  list  automatically.  Otherwise,  with  the  limitations  of  the  MLO  <->  CC 
context  switching  and  having  to  search  through  a  complicated  list  of  code  numbers,  the 
radiologists  will  become  frustrated,  which  might  impact  the  results  of  that  portion  of 
the  ROC  test. 
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Image  Processing: 

In  order  to  emulate  the  functionality  of  the  hot  lamp  and  the  magnifying  glass,  image 
processing  functions  will  be  implemented  on  the  digital  viewing  station.  However,  the 
zooming  functionality  included  with  the  E-systems  MegaScan  monitors  looks  overly 
complicated  for  a  function  that  the  radiologists  used  about  96%  of  the  time  in  the  analog 
portions  of  the  ROC  tests. 

With  regards  to  grayscale  modifications  of  the  digital  mammograms,  the  user  can 
change  both  the  brightness  and  the  contrast.  This  is  accomplished  fairly  easily  using  the 
mouse,  moving  it  either  up  or  down  for  contrast  modification  and  left  to  right  for 
brightness/ darkness  changes.  However,  as  there  are  three  buttons  on  the  optical 
mouse,  a  specific  series  of  button  pushes  are  necessary  to  invoke  and  dismiss  the 
grayscale  look-up  table  modification  software.  The  radiologists  are  going  to  have  to 
have  something  simple  to  get  through  the  set  of  50  image  cases  in  a  reasonable  amount 
of  time.  I  can  foresee  a  great  amount  of  frustration  with  the  current  user  interface  for 
zooming  and  look-up  table  modification.  All  but  one  of  the  mouse  buttons  should  be 
disabled  for  the  ROC  testing. 


Please  let  me  know  if  there  is  anything  else  that  you  may  require  in  this  matter.  It  has 
been  a  pleasure  working  with  you  and  Dr.  de  Parades  on  the  telemammography 
project. 

Sincerely, 


Brent  K.  Stewart,  Ph.D. 

Consultant  to  the  US  Army  Medical  Research  and  Development  Command  Grant 
Evaluation  of  a  Digital  Telemammography  System:  a  Model  for  a  Regional  System 
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